2025-09-11 11:59:01.362527] [error] [55289] [lcal_runner.cpp:79] commDomain must be a number or in id:size format, got: hccl_world_group_sub_1 [2025-09-11 11:59:01.362618] [error] [55289] [lcal_runner.cpp:99] AllReduceLcclRunner:0 Invalid commDomain: -1 [2025-09-11 11:59:01.362645] [error] [55289] [comm_pool.h:42] CommPool commCreateFunc fail [2025-09-11 11:59:01.362661] [error] [55289] [lcal_runner.cpp:46] AllReduceLcclRunner:0 get lcal comm from comm pool failed, rank : 1 [2025-09-11 11:59:01.362678] [error] [55289] [lccl_runner.cpp:28] AllReduceLcclRunner:0 GetLcalComm failed, rank: 1 [2025-09-11 11:59:01.362956] [error] [57135] [all_reduce_lccl_runner.cpp:31] lccl_ is null, rank: 1 [2025-09-11 11:59:01.363001] [error] [57135] [runner.cpp:132] AllReduceLcclRunner_2_0_4_1:1 Execute Failed. st: 28 [2025-09-11 11:59:01.363028] [error] [57135] [graph_runner.cpp:976] LinearRowParallelNoAddRunner_2_0_4:0 node[1] execute fail, runner name:AllReduceLcclRunner [2025-09-11 11:59:01.363046] [error] [57135] [runner.cpp:132] LinearRowParallelNoAddRunner_2_0_4:1 Execute Failed. st: 28 [2025-09-11 11:59:01.363061] [error] [57135] [graph_runner.cpp:976] AttentionRunner_2_0:0 node[4] execute fail, runner name:LinearRowParallelNoAddRunner [2025-09-11 11:59:01.363077] [error] [57135] [runner.cpp:132] AttentionRunner_2_0:1 Execute Failed. st: 28 [2025-09-11 11:59:01.363091] [error] [57135] [graph_runner.cpp:976] Prefill_layerRunner_2:0 node[0] execute fail, runner name:AttentionRunner [2025-09-11 11:59:01.363106] [error] [57135] [runner.cpp:132] Prefill_layerRunner_2:1 Execute Failed. st: 28 [2025-09-11 11:59:01.363120] [error] [57135] [operation_base.cpp:1023] Prefill_layer_2 execute Prefill_layerRunner fail [2025-09-11 11:59:01.363139] [error] [57135] [operation_base.cpp:1100] Prefill_layer_2 Launch fail, error code: 28 [2025-09-11 11:59:01.363] [55142] [281468921180512] [llmmodels] [ERROR] [model.cpp:546] Execute node[2] fail, error code: 28 [2025-09-11 11:59:01.364727] [error] [55296] [lcal_runner.cpp:79] commDomain must be a number or in id:size format, got: hccl_world_group_sub_1 [2025-09-11 11:59:01.364805] [error] [55296] [lcal_runner.cpp:99] AllReduceLcclRunner:0 Invalid commDomain: -1 [2025-09-11 11:59:01.364826] [error] [55296] [comm_pool.h:42] CommPool commCreateFunc fail [2025-09-11 11:59:01.364839] [error] [55296] [lcal_runner.cpp:46] AllReduceLcclRunner:0 get lcal comm from comm pool failed, rank : 2 [2025-09-11 11:59:01.364855] [error] [55296] [lccl_runner.cpp:28] AllReduceLcclRunner:0 GetLcalComm failed, rank: 2 [2025-09-11 11:59:01.365144] [error] [56926] [all_reduce_lccl_runner.cpp:31] lccl_ is null, rank: 2 [2025-09-11 11:59:01.365189] [error] [56926] [runner.cpp:132] AllReduceLcclRunner_2_0_4_1:1 Execute Failed. st: 28 [2025-09-11 11:59:01.365211] [error] [56926] [graph_runner.cpp:976] LinearRowParallelNoAddRunner_2_0_4:0 node[1] execute fail, runner name:AllReduceLcclRunner [2025-09-11 11:59:01.365228] [error] [56926] [runner.cpp:132] LinearRowParallelNoAddRunner_2_0_4:1 Execute Failed. st: 28 [2025-09-11 11:59:01.365242] [error] [56926] [graph_runner.cpp:976] AttentionRunner_2_0:0 node[4] execute fail, runner name:LinearRowParallelNoAddRunner [2025-09-11 11:59:01.365258] [error] [56926] [runner.cpp:132] AttentionRunner_2_0:1 Execute Failed. st: 28 [2025-09-11 11:59:01.365270] [error] [56926] [graph_runner.cpp:976] Prefill_layerRunner_2:0 node[0] execute fail, runner name:AttentionRunner [2025-09-11 11:59:01.365284] [error] [56926] [runner.cpp:132] Prefill_layerRunner_2:1 Execute Failed. st: 28 [2025-09-11 11:59:01.365297] [error] [56926] [operation_base.cpp:1023] Prefill_layer_2 execute Prefill_layerRunner fail [2025-09-11 11:59:01.365310] [error] [56926] [operation_base.cpp:1100] Prefill_layer_2 Launch fail, error code: 28 [2025-09-11 11:59:01.365] [55152] [281469138235744] [llmmodels] [ERROR] [model.cpp:546] Execute node[2] fail, error code: 28 [2025-09-11 11:59:01.368] [55142] [281471235912032] [llmmodels] [ERROR] [model.cpp:414] [MIE05E000002] Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.

[2025-09-11 11:59:01.368435] [error] [57135] [all_reduce_lccl_runner.cpp:31] lccl_ is null, rank: 1 [2025-09-11 11:59:01.368473] [error] [57135] [runner.cpp:132] AllReduceLcclRunner_3_0_4_1:1 Execute Failed. st: 28 [2025-09-11 11:59:01.368494] [error] [57135] [graph_runner.cpp:976] LinearRowParallelNoAddRunner_3_0_4:0 node[1] execute fail, runner name:AllReduceLcclRunner [2025-09-11 11:59:01.368511] [error] [57135] [runner.cpp:132] LinearRowParallelNoAddRunner_3_0_4:1 Execute Failed. st: 28 [2025-09-11 11:59:01.368527] [error] [57135] [graph_runner.cpp:976] AttentionRunner_3_0:0 node[4] execute fail, runner name:LinearRowParallelNoAddRunner [2025-09-11 11:59:01.368542] [error] [57135] [runner.cpp:132] AttentionRunner_3_0:1 Execute Failed. st: 28 [2025-09-11 11:59:01.368554] [error] [57135] [graph_runner.cpp:976] Prefill_layerRunner_3:0 node[0] execute fail, runner name:AttentionRunner [2025-09-11 11:59:01.368568] [error] [57135] [runner.cpp:132] Prefill_layerRunner_3:1 Execute Failed. st: 28 [2025-09-11 11:59:01.368582] [error] [57135] [operation_base.cpp:1023] Prefill_layer_3 execute Prefill_layerRunner fail [2025-09-11 11:59:01.368596] [error] [57135] [operation_base.cpp:1100] Prefill_layer_3 Launch fail, error code: 28 [2025-09-11 11:59:01.368] [55142] [281468921180512] [llmmodels] [ERROR] [model.cpp:546] Execute node[3] fail, error code: 28 [2025-09-11 11:59:01.369] [55152] [281471458144608] [llmmodels] [ERROR] [model.cpp:414] [MIE05E000002] Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.

[2025-09-11 11:59:01.370112] [error] [56926] [all_reduce_lccl_runner.cpp:31] lccl_ is null, rank: 2 [2025-09-11 11:59:01.370145] [error] [56926] [runner.cpp:132] AllReduceLcclRunner_3_0_4_1:1 Execute Failed. st: 28 [2025-09-11 11:59:01.370164] [error] [56926] [graph_runner.cpp:976] LinearRowParallelNoAddRunner_3_0_4:0 node[1] execute fail, runner name:AllReduceLcclRunner [2025-09-11 11:59:01.370180] [error] [56926] [runner.cpp:132] LinearRowParallelNoAddRunner_3_0_4:1 Execute Failed. st: 28 [2025-09-11 11:59:01.370194] [error] [56926] [graph_runner.cpp:976] AttentionRunner_3_0:0 node[4] execute fail, runner name:LinearRowParallelNoAddRunner [2025-09-11 11:59:01.370208] [error] [56926] [runner.cpp:132] AttentionRunner_3_0:1 Execute Failed. st: 28 [2025-09-11 11:59:01.370222] [error] [56926] [graph_runner.cpp:976] Prefill_layerRunner_3:0 node[0] execute fail, runner name:AttentionRunner [2025-09-11 11:59:01.370234] [error] [56926] [runner.cpp:132] Prefill_layerRunner_3:1 Execute Failed. st: 28 [2025-09-11 11:59:01.370247] [error] [56926] [operation_base.cpp:1023] Prefill_layer_3 execute Prefill_layerRunner fail [2025-09-11 11:59:01.370263] [error] [56926] [operation_base.cpp:1100] Prefill_layer_3 Launch fail, error code: 28 [2025-09-11 11:59:01.370] [55152] [281469138235744] [llmmodels] [ERROR] [model.cpp:546] Execute node[3] fail, error code: 28 [2025-09-11 11:59:01.011+0800] [55152] [281473449849184] [batchscheduler] [ERROR] [model.py:61] : [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document. Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 59, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 299, in init self.cache_manager = self.warm_up( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 456, in warm_up npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 626, in __warmup_standard npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 614, in __warmup_prefill npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 708, in __auto_warmup self.__execute_warm_up(cache_manager, input_metadata, dummy=True) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 508, in __execute_warm_up raise e File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 499, in __execute_warm_up self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode, File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 570, in _warm_up super()._warm_up(model_inputs, **kwargs) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 247, in _warm_up logits = self.forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward logits = self._forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs result = self.forward_tensor( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor result = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward res = self.model.forward(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 536, in forward logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 467, in execute_ascend_operator acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.

[2025-09-11 11:59:01.011+0800] [55152] [281473449849184] [batchscheduler] [ERROR] [model.py:64] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'} [2025-09-11 11:59:01.011+0800] [55142] [281473223422304] [batchscheduler] [ERROR] [model.py:61] : [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document. Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 59, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 299, in init self.cache_manager = self.warm_up( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 456, in warm_up npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 626, in __warmup_standard npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 614, in __warmup_prefill npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 708, in __auto_warmup self.__execute_warm_up(cache_manager, input_metadata, dummy=True) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 508, in __execute_warm_up raise e File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 499, in __execute_warm_up self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode, File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 570, in _warm_up super()._warm_up(model_inputs, **kwargs) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 247, in _warm_up logits = self.forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward logits = self._forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward result = self.forward_from_model_inputs(model_inputs, npu_cache, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs result = self.forward_tensor( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor result = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward res = self.model.forward( kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 536, in forward logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 467, in execute_ascend_operator acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.