我尝试使用MindYOLO, 训练改进后的YOLOv8模型，我在模型中添加了一些自己的模块，然后在静态图编译阶段会报错，报错日志如下，请问是哪里出了问题呢
[ERROR] GE_ADPT(165035,ffff8f1b6010,python):2025-01-03-16:27:53.024.504 [mindspore/ccsrc/transform/graph_ir/graph_runner.cc:444] CompileGraph] Call GE CompileGraph Failed, ret is: 1343225857
Traceback (most recent call last):
File "/home/ma-user/work/mindyolo/train.py", line 338, in <module>
train(args)
File "/home/ma-user/work/mindyolo/train.py", line 293, in train
trainer.train(
File "/home/ma-user/work/mindyolo/mindyolo/utils/trainer_factory.py", line 170, in train
run_context.loss, run_context.lr = self.train_step(imgs, labels, segments,
File "/home/ma-user/work/mindyolo/mindyolo/utils/trainer_factory.py", line 366, in train_step
loss, loss_item, _, grads_finite = self.train_step_fn(imgs, labels, True)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 941, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, dyn_args, process_obj, jit_config)(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 185, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 566, in __call__
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 563, in __call__
phase = self.compile(self.fn.__name__, *args_list, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 654, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, kwargs, phase, True)
RuntimeError: Compile graph kernel_graph1 failed.
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E40024: 2025-01-03-16:26:50.283.451 Failed call Python Func/Meathod [call_op_func], Reason[Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te_fusion/fusion_manager.py", line 816, in call_op_func
return op_func(*inputs, *outputs, *new_attrs)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/concat_d.py", line 60, in op_select_format
return concat_v2_op_select_format(input_values, output_data, concat_dim, kernel_name)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_v2_d.py", line 140, in op_select_format
if input_shape[concat_dim] % align_len != 0:
IndexError: list index out of range
] [THREAD:165035]
Possible Cause: The Python Func/Meathod does not exist.
TraceBack (most recent call last):
Failed call Python Func/Meathod [call_op_func], Reason[Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te_fusion/fusion_manager.py", line 816, in call_op_func
return op_func(*inputs, *outputs, *new_attrs)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_d.py", line 62, in op_select_format
return concat_v2_op_select_format(input_values, output_data, concat_dim, kernel_name)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_v2_d.py", line 140, in op_select_format
if input_shape[concat_dim] % align_len != 0:
IndexError: list index out of range
] [THREAD:165035]
Optype [Pooling] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type Pooling is not found in this op store.[tbe-custom1]:op type Pooling is not found in this op store.[tbe-custom2]:op type Pooling is not found in this op store.[tbe-custom3]:op type Pooling is not found in this op store.[tbe-custom4]:op type Pooling is not found in this op store.[tbe-custom5]:op type Pooling is not found in this op store.[tbe-custom]:op type Pooling is not found in this op store.[tbe-custom1]:op type Pooling is not found in this op store.[tbe-custom2]:op type Pooling is not found in this op store.[tbe-custom3]:op type Pooling is not found in this op store.[tbe-custom4]:op type Pooling is not found in this op store.[tbe-custom5]:op type Pooling is not found in this op store.[Static shape check]:data type DT_FLOAT of input [x] is not supported. All supported data type and format of tensor input0.x is: Data Type: {DT_FLOAT16,DT_INT8}Format:{NC1HWC0,NC1HWC0}.[THREAD:165035]
No supported Ops kernel and engine are found for [Default/AdaptiveMaxPool2D-op0_pooling], optype [Pooling].[THREAD:165035]
[Call][PreRun] Failed, graph_id:2, session_id:0.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4409][THREAD:165035]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:0, graph_id:2.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1165][THREAD:165035]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_graph_executor.cc:988 CompileGraph
点赞
收藏
回复
分享
举报
已解决
我尝试使用MindYOLO, 训练改进后的YOLOv8模型，我在模型中添加了一些自己的模块，然后在静态图编译阶段会报错，报错日志如下，请问是哪里出了问题呢
[ERROR] GE_ADPT(165035,ffff8f1b6010,python):2025-01-03-16:27:53.024.504 [mindspore/ccsrc/transform/graph_ir/graph_runner.cc:444] CompileGraph] Call GE CompileGraph Failed, ret is: 1343225857
Traceback (most recent call last):
File "/home/ma-user/work/mindyolo/train.py", line 338, in <module>
train(args)
File "/home/ma-user/work/mindyolo/train.py", line 293, in train
trainer.train(
File "/home/ma-user/work/mindyolo/mindyolo/utils/trainer_factory.py", line 170, in train
run_context.loss, run_context.lr = self.train_step(imgs, labels, segments,
File "/home/ma-user/work/mindyolo/mindyolo/utils/trainer_factory.py", line 366, in train_step
loss, loss_item, _, grads_finite = self.train_step_fn(imgs, labels, True)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 941, in staging_specialize
out = _MindsporeFunctionExecutor(func, hash_obj, dyn_args, process_obj, jit_config)(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 185, in wrapper
results = fn(*arg, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 566, in __call__
raise err
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 563, in __call__
phase = self.compile(self.fn.__name__, *args_list, **kwargs)
File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 654, in compile
is_compile = self._graph_executor.compile(self.fn, compile_args, kwargs, phase, True)
RuntimeError: Compile graph kernel_graph1 failed.
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E40024: 2025-01-03-16:26:50.283.451 Failed call Python Func/Meathod [call_op_func], Reason[Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te_fusion/fusion_manager.py", line 816, in call_op_func
return op_func(*inputs, *outputs, *new_attrs)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/dynamic/concat_d.py", line 60, in op_select_format
return concat_v2_op_select_format(input_values, output_data, concat_dim, kernel_name)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_v2_d.py", line 140, in op_select_format
if input_shape[concat_dim] % align_len != 0:
IndexError: list index out of range
] [THREAD:165035]
Possible Cause: The Python Func/Meathod does not exist.
TraceBack (most recent call last):
Failed call Python Func/Meathod [call_op_func], Reason[Traceback (most recent call last):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te_fusion/fusion_manager.py", line 816, in call_op_func
return op_func(*inputs, *outputs, *new_attrs)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_d.py", line 62, in op_select_format
return concat_v2_op_select_format(input_values, output_data, concat_dim, kernel_name)
File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/concat_v2_d.py", line 140, in op_select_format
if input_shape[concat_dim] % align_len != 0:
IndexError: list index out of range
] [THREAD:165035]
Optype [Pooling] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type Pooling is not found in this op store.[tbe-custom1]:op type Pooling is not found in this op store.[tbe-custom2]:op type Pooling is not found in this op store.[tbe-custom3]:op type Pooling is not found in this op store.[tbe-custom4]:op type Pooling is not found in this op store.[tbe-custom5]:op type Pooling is not found in this op store.[tbe-custom]:op type Pooling is not found in this op store.[tbe-custom1]:op type Pooling is not found in this op store.[tbe-custom2]:op type Pooling is not found in this op store.[tbe-custom3]:op type Pooling is not found in this op store.[tbe-custom4]:op type Pooling is not found in this op store.[tbe-custom5]:op type Pooling is not found in this op store.[Static shape check]:data type DT_FLOAT of input [x] is not supported. All supported data type and format of tensor input0.x is: Data Type: {DT_FLOAT16,DT_INT8}Format:{NC1HWC0,NC1HWC0}.[THREAD:165035]
No supported Ops kernel and engine are found for [Default/AdaptiveMaxPool2D-op0_pooling], optype [Pooling].[THREAD:165035]
[Call][PreRun] Failed, graph_id:2, session_id:0.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4409][THREAD:165035]
[Compile][Graph]Compile graph failed, error code:1343225857, session_id:0, graph_id:2.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1165][THREAD:165035]
(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_graph_executor.cc:988 CompileGraph