Traceback (most recent call last):
File "/home/ma-user/work/code/train.py", line 1053, in <module>
main(opt)
File "/home/ma-user/work/code/train.py", line 921, in main
train(opt.hyp, opt, device, callbacks)
File "/home/ma-user/work/code/train.py", line 609, in train
results, maps, _ = validate.run(
File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ma-user/work/code/val.py", line 227, in run
scale_boxes(im[si].shape[1:], predn[:, :4], shape, shapes[si][1]) # native-space pred
File "/home/ma-user/work/code/utils/general.py", line 834, in scale_boxes
boxes[..., [1, 3]] -= pad[1] # y padding
RuntimeError: ACL stream synchronize failed, error code:507034
[W NPUStream.cpp:382] Warning: NPU warning, error code is 507034[Error]:
[Error]: Vector core execution timed out.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[vector core timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-03-18-19:22:52.255.888 wait for compute device to finish failed, runtime result = 507034.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507034[Error]:
[Error]: Vector core execution timed out.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[vector core timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-03-18-19:22:52.793.077 wait for compute device to finish failed, runtime result = 507034.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
train的时候没有问题，但是一轮训练结束之后，val的时候会报错，该怎么解决呀？
点赞
收藏
回复
分享
举报
已解决
File "/home/ma-user/work/code/train.py", line 1053, in <module>
main(opt)
File "/home/ma-user/work/code/train.py", line 921, in main
train(opt.hyp, opt, device, callbacks)
File "/home/ma-user/work/code/train.py", line 609, in train
results, maps, _ = validate.run(
File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ma-user/work/code/val.py", line 227, in run
scale_boxes(im[si].shape[1:], predn[:, :4], shape, shapes[si][1]) # native-space pred
File "/home/ma-user/work/code/utils/general.py", line 834, in scale_boxes
boxes[..., [1, 3]] -= pad[1] # y padding
RuntimeError: ACL stream synchronize failed, error code:507034
[W NPUStream.cpp:382] Warning: NPU warning, error code is 507034[Error]:
[Error]: Vector core execution timed out.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[vector core timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-03-18-19:22:52.255.888 wait for compute device to finish failed, runtime result = 507034.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
(function npuSynchronizeUsedDevices)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507034[Error]:
[Error]: Vector core execution timed out.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronize execute failed, reason=[vector core timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2025-03-18-19:22:52.793.077 wait for compute device to finish failed, runtime result = 507034.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
TraceBack (most recent call last):
train的时候没有问题，但是一轮训练结束之后，val的时候会报错，该怎么解决呀？