MindIE 1.0.0
CANN 8.0.0.beta1
Driver 24.1.0.1
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.076 [dnnengine_manager.cc:407]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Check][OpSupported]Op type TransData of ops kernel AIcoreEngine is unsupported, reason : [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.107 [dnnengine_manager.cc:407]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Check][OpSupported]Op type TransData of ops kernel aicpu_ascend_kernel is unsupported, reason : Transdata op, groups should be greater than 1, but now is 1
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.268 [dnnengine_manager.cc:422]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Get][DNNEngineName]Can't find any supported ops kernel and engine of trans_TransData_0, type is TransData
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.295 [engine_place.cc:67]6922 SelectEngine: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][SUB_OPT][Check][Param] Can not find engine of op name trans_TransData_0 type TransData
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.338 [engine_place.cc:148]6922 operator(): ErrorNo: 4294967295(failed) [COMP][SUB_OPT]Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.361 [engine_place.cc:169]6364 Run: ErrorNo: 1343225857(Parameter invalid!) [COMP][SUB_OPT]Failed to SelectEngine.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.801 [engine_place.cc:122]6364 RunAllSubgraphs: ErrorNo: 4294967295(failed) [COMP][SUB_OPT]RunAllSubgraphs failed, graph=online.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.823 [graph_manager.cc:3310]6364 OptimizeSubgraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][Run] Engine placer run failed, graph:online.
[INFO] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.830 [graph_manager.cc:917]6364 PreRunOptimizeSubGraph:[GEPERFTRACE] The time cost of GraphManager::OptimizeSubgraph is [3243] micro second.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.836 [graph_manager.cc:917]6364 PreRunOptimizeSubGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Process][GraphManager_OptimizeSubgraph] failed
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.841 [graph_manager.cc:3837]6364 OptimizeGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Run][PreRunOptimizeSubGraph] failed for graph:online, session_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.848 [pne_model_builder.cc:120]6364 OptimizeGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Optimize][Graph] failed, graph = online, engine = NPU
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.864 [graph_manager.cc:1076]6364 PreRun: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Build][Model] failed, session_id:0, graph_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.453 [graph_manager.cc:1133]6364 StartForRunGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][PreRun] Failed, graph_id:0, session_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.465 [graph_manager.cc:1734]6364 BuildGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][StartForRunGraph] failed! graph_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.516 [ge_generator.cc:1619]6364 BuildModelWithGraphId: ErrorNo: 1343266819(Graph manager build graph failed.) [COMP][DEFAULT][Build][Graph] fail, graph id: 0
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.554 [local_compiler.cpp:120]6364 OnlineCompileAndDump: [COMP][DEFAULT][Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.588 [local_compiler.cpp:133]6364 OnlineCompileAndDump: [COMP][DEFAULT]OnlineCompileAndDump failed
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.793 [acl_resource_manager.cpp:525]6364 GetOpModel: [FINAL][FINAL][Build][Op]Fail to build op model
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.831 [op_compiler.cpp:196]6364 aclopCompileAndExecute: [FINAL][FINAL]build op model failed, result = 500002
[INFO] RUNTIME(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.950 [api_impl.cc:6836] 6364 PeekLastErr: level=0 err=0.
[INFO] RUNTIME(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.958 [api_impl.cc:6836] 6364 PeekLastErr: level=0 err=0.
2025-04-14 07:23:23,120 [ERROR] model.py:39 - [Model] >>> Exception:InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002
[ERROR] 2025-04-14-07:23:23 (PID:5989, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3002: [PID: 5989] 2025-04-14-07:23:23.113.037 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
File "/usr/local/lib/python3.10/dist-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 452, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 150, in init_ascend_weight
weight_wrapper = self.get_weights()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in get_weights
weight_wrapper.register_layer(layer, self.quantize)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 242, in register_layer
self.register_layer_attn(layer, self.attn_wrapper, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 168, in register_layer_attn
self.register_layer_linear_pack(layer, wrapper, quantize_type, 'attn')
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 133, in register_layer_linear_pack
self.register_linear_wrapper(linear, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 104, in register_linear_wrapper
self.register_linear_bias(linear)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 67, in register_linear_bias
linear.weight.data = self.weight_format_cast(linear.weight.data, enable_nz=enable_nz)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 59, in weight_format_cast
torch_npu.npu_format_cast_(tensor, 29)
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 692, in __call__
return self._op(*args, **kwargs or {})
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002
[ERROR] 2025-04-14-07:23:23 (PID:5989, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3002: [PID: 5989] 2025-04-14-07:23:23.113.037 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
点赞
收藏
回复
分享
举报
已解决
MindIE 1.0.0
CANN 8.0.0.beta1
Driver 24.1.0.1
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.076 [dnnengine_manager.cc:407]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Check][OpSupported]Op type TransData of ops kernel AIcoreEngine is unsupported, reason : [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.107 [dnnengine_manager.cc:407]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Check][OpSupported]Op type TransData of ops kernel aicpu_ascend_kernel is unsupported, reason : Transdata op, groups should be greater than 1, but now is 1
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.268 [dnnengine_manager.cc:422]6922 GetDNNEngineName: ErrorNo: 1343242282(assign engine failed) [COMP][SUB_OPT][Get][DNNEngineName]Can't find any supported ops kernel and engine of trans_TransData_0, type is TransData
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.295 [engine_place.cc:67]6922 SelectEngine: ErrorNo: 1343229963(GE is not yet initialized or is finalized.) [COMP][SUB_OPT][Check][Param] Can not find engine of op name trans_TransData_0 type TransData
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.338 [engine_place.cc:148]6922 operator(): ErrorNo: 4294967295(failed) [COMP][SUB_OPT]Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.361 [engine_place.cc:169]6364 Run: ErrorNo: 1343225857(Parameter invalid!) [COMP][SUB_OPT]Failed to SelectEngine.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.801 [engine_place.cc:122]6364 RunAllSubgraphs: ErrorNo: 4294967295(failed) [COMP][SUB_OPT]RunAllSubgraphs failed, graph=online.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.823 [graph_manager.cc:3310]6364 OptimizeSubgraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][Run] Engine placer run failed, graph:online.
[INFO] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.830 [graph_manager.cc:917]6364 PreRunOptimizeSubGraph:[GEPERFTRACE] The time cost of GraphManager::OptimizeSubgraph is [3243] micro second.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.836 [graph_manager.cc:917]6364 PreRunOptimizeSubGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Process][GraphManager_OptimizeSubgraph] failed
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.841 [graph_manager.cc:3837]6364 OptimizeGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Run][PreRunOptimizeSubGraph] failed for graph:online, session_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.848 [pne_model_builder.cc:120]6364 OptimizeGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Optimize][Graph] failed, graph = online, engine = NPU
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.113.864 [graph_manager.cc:1076]6364 PreRun: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Build][Model] failed, session_id:0, graph_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.453 [graph_manager.cc:1133]6364 StartForRunGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][PreRun] Failed, graph_id:0, session_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.465 [graph_manager.cc:1734]6364 BuildGraph: ErrorNo: 4294967295(failed) [COMP][SUB_OPT][Call][StartForRunGraph] failed! graph_id:0.
[ERROR] GE(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.516 [ge_generator.cc:1619]6364 BuildModelWithGraphId: ErrorNo: 1343266819(Graph manager build graph failed.) [COMP][DEFAULT][Build][Graph] fail, graph id: 0
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.554 [local_compiler.cpp:120]6364 OnlineCompileAndDump: [COMP][DEFAULT][Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.588 [local_compiler.cpp:133]6364 OnlineCompileAndDump: [COMP][DEFAULT]OnlineCompileAndDump failed
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.793 [acl_resource_manager.cpp:525]6364 GetOpModel: [FINAL][FINAL][Build][Op]Fail to build op model
[ERROR] ASCENDCL(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.831 [op_compiler.cpp:196]6364 aclopCompileAndExecute: [FINAL][FINAL]build op model failed, result = 500002
[INFO] RUNTIME(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.950 [api_impl.cc:6836] 6364 PeekLastErr: level=0 err=0.
[INFO] RUNTIME(5989,mindie_llm_backend_connector):2025-04-14-07:23:23.114.958 [api_impl.cc:6836] 6364 PeekLastErr: level=0 err=0.
2025-04-14 07:23:23,120 [ERROR] model.py:39 - [Model] >>> Exception:InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002
[ERROR] 2025-04-14-07:23:23 (PID:5989, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3002: [PID: 5989] 2025-04-14-07:23:23.113.037 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
File "/usr/local/lib/python3.10/dist-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
File "/usr/local/lib/python3.10/dist-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 452, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 150, in init_ascend_weight
weight_wrapper = self.get_weights()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 136, in get_weights
weight_wrapper.register_layer(layer, self.quantize)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 242, in register_layer
self.register_layer_attn(layer, self.attn_wrapper, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 168, in register_layer_attn
self.register_layer_linear_pack(layer, wrapper, quantize_type, 'attn')
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 133, in register_layer_linear_pack
self.register_linear_wrapper(linear, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 104, in register_linear_wrapper
self.register_linear_bias(linear)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 67, in register_linear_bias
linear.weight.data = self.weight_format_cast(linear.weight.data, enable_nz=enable_nz)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 59, in weight_format_cast
torch_npu.npu_format_cast_(tensor, 29)
File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 692, in __call__
return self._op(*args, **kwargs or {})
RuntimeError: InnerRun:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:200 OPS function error: Identity, error code is 500002
[ERROR] 2025-04-14-07:23:23 (PID:5989, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[Error]: A GE error occurs in the system.
Rectify the fault based on the error information in the ascend log.
EZ3002: [PID: 5989] 2025-04-14-07:23:23.113.037 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom2]:op type TransData is not found in this op store.[tbe-custom3]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]