[2025-03-18 00:11:08,458] [4133] [281473154927168] [llm] [INFO] [model_test.py-739] : Total passed cases: 0
[2025-03-18 00:11:08,458] [4133] [281473154927168] [llm] [INFO] [model_test.py-740] : Failed case details: batch_size: 16, seq_len_in: 256, seq_len_out: 256
[2025-03-18 00:11:08,460] [4133] [281473154927168] [llm] [INFO] [model_test.py-845] : performance test end
[2025-03-18 00:11:10.432866] [error] [4134] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config
[2025-03-18 00:11:10.432948] [error] [4134] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-03-18 00:11:10.432959] [error] [4134] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:6
[2025-03-18 00:11:10.435294] [error] [4134] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config
[2025-03-18 00:11:10.435319] [error] [4134] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-03-18 00:11:10.435332] [error] [4134] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:6
[2025-03-18 00:11:10.464506] [error] [5730] [all_reduce_hccl_runner.cpp:38] hcclComm is null, rank: 6
[2025-03-18 00:11:10.464543] [error] [5730] [runner.cpp:115] AllReduceHcclRunner_2_0_18_1:1 Execute Failed. st: 3
[2025-03-18 00:11:10.464557] [error] [5730] [graph_runner.cpp:906] LinearRowParallelNoAddRunner_2_0_18:0  node[1] execute fail, runner name:AllReduceHcclRunner
[2025-03-18 00:11:10.464568] [error] [5730] [runner.cpp:115] LinearRowParallelNoAddRunner_2_0_18:1 Execute Failed. st: 3
[2025-03-18 00:11:10.464577] [error] [5730] [graph_runner.cpp:906] AttentionRunner_2_0:0  node[18] execute fail, runner name:LinearRowParallelNoAddRunner
[2025-03-18 00:11:10.464587] [error] [5730] [runner.cpp:115] AttentionRunner_2_0:1 Execute Failed. st: 3
[2025-03-18 00:11:10.464597] [error] [5730] [graph_runner.cpp:906] Prefill_layerRunner_2:0  node[0] execute fail, runner name:AttentionRunner
[2025-03-18 00:11:10.464607] [error] [5730] [runner.cpp:115] Prefill_layerRunner_2:1 Execute Failed. st: 3
[2025-03-18 00:11:10.464616] [error] [5730] [operation_base.cpp:726] Prefill_layer_2 execute Prefill_layerRunner fail
[2025-03-18 00:11:10.464627] [error] [5730] [operation_base.cpp:764] Prefill_layer_2 execute fail, error code: 3
[2025-03-18 00:11:10.465419] [error] [4134] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config
[2025-03-18 00:11:10.465443] [error] [4134] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-03-18 00:11:10.465456] [error] [4134] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:6
[2025-03-18 00:11:10.467885] [error] [4134] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config
[2025-03-18 00:11:10.467910] [error] [4134] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-03-18 00:11:10.467919] [error] [4134] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:6
[2025-03-18 00:11:10.470424] [error] [5730] [all_reduce_hccl_runner.cpp:38] hcclComm is null, rank: 6
[2025-03-18 00:11:10.470448] [error] [5730] [runner.cpp:115] AllReduceHcclRunner_3_0_18_1:1 Execute Failed. st: 3
[2025-03-18 00:11:10.470461] [error] [5730] [graph_runner.cpp:906] LinearRowParallelNoAddRunner_3_0_18:0  node[1] execute fail, runner name:AllReduceHcclRunner
[2025-03-18 00:11:10.470471] [error] [5730] [runner.cpp:115] LinearRowParallelNoAddRunner_3_0_18:1 Execute Failed. st: 3
[2025-03-18 00:11:10.470482] [error] [5730] [graph_runner.cpp:906] AttentionRunner_3_0:0  node[18] execute fail, runner name:LinearRowParallelNoAddRunner
[2025-03-18 00:11:10.470491] [error] [5730] [runner.cpp:115] AttentionRunner_3_0:1 Execute Failed. st: 3
[2025-03-18 00:11:10.470500] [error] [5730] [graph_runner.cpp:906] Prefill_layerRunner_3:0  node[0] execute fail, runner name:AttentionRunner
[2025-03-18 00:11:10.470509] [error] [5730] [runner.cpp:115] Prefill_layerRunner_3:1 Execute Failed. st: 3
[2025-03-18 00:11:10.470516] [error] [5730] [operation_base.cpp:726] Prefill_layer_3 execute Prefill_layerRunner fail
[2025-03-18 00:11:10.470526] [error] [5730] [operation_base.cpp:764] Prefill_layer_3 execute fail, error code: 3
[2025-03-18 00:11:10,474] [4134] [281473322838592] [llm] [ERROR] [logging.py-50] : Error caught during Warm-up: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document. 

[2025-03-18 00:11:10,474] [4134] [281473322838592] [llm] [ERROR] [logging.py-50] : Warm-up failed.
[2025-03-18 00:11:10,474] [4134] [281473322838592] [llm] [ERROR] [logging.py-50] : If due to out of memory error, try to remove the largest memory-consuming case_pair: [0, 0] and batch_size: 0 combination.
[2025-03-18 00:11:10,477] [4134] [281473322838592] [llm] [ERROR] [logging.py-50] : Error caught during inference: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document. 

[2025-03-18 00:11:10,477] [4134] [281473322838592] [llm] [ERROR] [logging.py-50] : Stack trace:
Traceback (most recent call last):
  File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 707, in run_performance_test
    raise e
  File "/usr/local/Ascend/atb-models/tests/modeltest/base/model_test.py", line 694, in run_performance_test
    self.pa_runner.warm_up()
  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
    generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
  File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
    req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 338, in generate_token_with_clocking
    req_finished = generate_token(model, cache_manager, input_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
    logits = model.forward(
             ^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
    return self.model.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 466, in forward
    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 418, in execute_ascend_operator
    acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document. 


[2025-03-18 00:11:10,477] [4134] [281473322838592] [llm] [INFO] [model_test.py-739] : Total passed cases: 0
[2025-03-18 00:11:10,477] [4134] [281473322838592] [llm] [INFO] [model_test.py-740] : Failed case details: batch_size: 16, seq_len_in: 256, seq_len_out: 256
[2025-03-18 00:11:10,479] [4134] [281473322838592] [llm] [INFO] [model_test.py-845] : performance test end
[2025-03-18 00:16:35,545] torch.distributed.elastic.agent.server.api: [ERROR] Error waiting on exit barrier. Elapsed: 300.10409569740295 seconds