300I DUO 进行yolov8训练,报下面的错误:
Starting training for 10 epochs...
Closing dataloader mosaic
Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size
0%| | 0/8 [00:00<?, ?it/s].....[W compiler_depend.ts:82] Warning: The oprator of add is executed, Currently High Accuracy but Low Performance OP with 64-bit has been used, Please Do Some Cast at Python Functions with 32-bit for Better Performance! (function operator())
0%| | 0/8 [01:34<?, ?it/s]
Traceback (most recent call last):
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/test.py", line 12, in <module>
results = model.train( data='/data/code_ws/yolov8/defect_detection.yaml', epochs=10, imgsz=640, batch=1, name='dataset/runs',device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/model.py", line 784, in train
self.trainer.train()
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/trainer.py", line 207, in train
self._do_train(world_size)
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/trainer.py", line 397, in _do_train
self.scaler.scale(self.loss).backward()
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/usr/local/lib64/python3.11/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:234 OPS function error: MaxPoolGradWithArgmaxV1, error code is 500002
[ERROR] 2025-05-07-16:30:29 (PID:315731, Device:4, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3003: [PID: 315731] 2025-05-07-16:30:29.698.158 No supported Ops kernel and engine are found for [MaxPoolGradWithArgmaxV1445], optype [MaxPoolGradWithArgmaxV1].
Possible Cause: The operator is not supported by the system. Therefore, no hit is found in any operator information library.
Solution: 1. Check that the OPP component is installed properly. 2. Submit an issue to request for the support of this operator type.
TraceBack (most recent call last):
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:444, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
300I DUO 进行yolov8训练,报下面的错误:
Starting training for 10 epochs...
Closing dataloader mosaic
Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size
0%| | 0/8 [00:00<?, ?it/s].....[W compiler_depend.ts:82] Warning: The oprator of add is executed, Currently High Accuracy but Low Performance OP with 64-bit has been used, Please Do Some Cast at Python Functions with 32-bit for Better Performance! (function operator())
0%| | 0/8 [01:34<?, ?it/s]
Traceback (most recent call last):
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/test.py", line 12, in <module>
results = model.train( data='/data/code_ws/yolov8/defect_detection.yaml', epochs=10, imgsz=640, batch=1, name='dataset/runs',device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/model.py", line 784, in train
self.trainer.train()
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/trainer.py", line 207, in train
self._do_train(world_size)
File "/data/repos_models/modelzoo/modelzoo-GPL-master/built-in/PyTorch/Official/cv/object_detection/Yolov8_for_PyTorch/ultralytics/engine/trainer.py", line 397, in _do_train
self.scaler.scale(self.loss).backward()
File "/usr/local/lib64/python3.11/site-packages/torch/_tensor.py", line 492, in backward
torch.autograd.backward(
File "/usr/local/lib64/python3.11/site-packages/torch/autograd/__init__.py", line 251, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:234 OPS function error: MaxPoolGradWithArgmaxV1, error code is 500002
[ERROR] 2025-05-07-16:30:29 (PID:315731, Device:4, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3003: [PID: 315731] 2025-05-07-16:30:29.698.158 No supported Ops kernel and engine are found for [MaxPoolGradWithArgmaxV1445], optype [MaxPoolGradWithArgmaxV1].
Possible Cause: The operator is not supported by the system. Therefore, no hit is found in any operator information library.
Solution: 1. Check that the OPP component is installed properly. 2. Submit an issue to request for the support of this operator type.
TraceBack (most recent call last):
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:444, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]