[2025-02-07 08:56:35,846] [61826] [281472101118176] [11m] [INFO][logging.py-227] 

soc_version=223, need_nz=False, matmul_nd_nz=False) 

NPUSocInfo(soc_name=' 

[2025-02-07 08:56:35,841] [61849] [281472329576672] [11m] [INFO][dist.py-81] : initialize_distributed has been Set 

[2025-02-07 08:56:55.239824] [error] [61946] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config 

[2025-02-07 08:56:55.239878] [error] [61946] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-02-07 08:56:55.239888] [error] [61946] [hccl_runner.cpp:72] AllReduceHcclRunner:® get hccl comm fail by rank:7 

[2025-02-07 08:56:55.240235] [error] [61946] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config 

[2025-02-07 08:56:55.240248] [error] [61946] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-02-07 08:56:55.240255] [error] [61946] [hccl_runner.cpp:72] AllReduceHcclRunner:6 get hccl comm fail by rank:7 

[2025-02-07 08:56:55.255618] [error] [61946] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config 

[2025-02-07 08:56:55.255640] [error] [61946] [comm_pool.h:43] CommPool commCreateFunc fail 

[2625-62-67 68:56:55.255644] [error] [61946] [hcct_runner.cpp:72] ALLReduceHcctRunner:6 get hccL comm fail by rank:7 

[2025-02-07 08:56:55.255998] [error] [61946] [hccl_runner.cpp:103] HCCL CommInitClusteriInfo ERROR1 should check rankTableFile config 

[2025-02-07 08:56:55.256012] [error] [61946] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-02-07 08:56:55.256018] [error] [61946] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:7 

[2025-02-07 08:56:55.256543] [error] [63583] [runner.cpp:115] AllReduceHcclRunner_2_0 18 1:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.256555] [error] [63583] [graph_runner.cpp:906] LinearRowParallelNoAddRunner_2 90 18:0 node[1] execute fail, runner name:AlLReduceHcclRunner 

[2025-02-07 08:56:55.256563] [error] [63583] [runner.cpp:115] LinearRowParallelNoAddRunner_2 0 18:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.256570] [error] [63583] [graph_runner.cpp:906] AttentionRunner_2_0:@ node[18] execute fail, runner name:LinearRowParallelNoAddRunner 

12025-02-07 08-56-55. 256578] [error] [63583] [runner.con:115] AttentionRunner 2 0-1 Execute Failed. st: 32 

[2025-02-07 08:56:55.256578] [error] [63583] [runner.cpp:115] AttentionRunner_2 0:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.256584] [error] [63583] [graph_runner.cpp:906] Prefill_layerRunner_2:9 node[0] execute fail, runner name:AttentionRunner 

[2025-02-07 08:56:55.256593] [error] [63583] [runner.cpp:115] Prefill_layerRunner_2:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.256601] [error] [63583] [operation_base.cpp:726] Prefill_layer_2 execute Prefill_layerRunner fail 

[2025-02-07 08:56:55.258421] [error] [63583] [all reduce_hccl_runner.cpp:38] hcclComm is null, rank: 7 

[2025-02-07 08:56:55.258437] [error] [63583] [runner.cpp:115] AllReduceHcclRunner_3_0 18 1:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.258447] [error] 63583] graph_runner.cpp:906] LinearRowParaTlelNoAddRunner_3_0_18:0  node[1] execute fail, runner name:AllReduceHcclRunner 

(2025-02-07 08:56:55.258455] [error] [63583] [runner.cpp:115] LinearRowParallelNoAddRunner_3 0 18:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.258463] [error] [63583] [graph_runner.cpp:906] AttentionRunner_3 0:0 node[18] execute fail, runner name:LinearRowParallelNoAddRunner 

[2025-02-07 08:56:55.258471] [error] [63583] [runner.cpp:115] AttentionRunner_3 0:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.258485] [error] [63583] [runner.cpp:115] Prefill_layerRunner_3:1 Execute Failed. st: 3 

[2025-02-07 08:56:55.258495] [error] [63583] [operation_base.cpp:726] Prefill_layer_3 execute Prefill_layerRunner fail 

[2025-02-07 08:56:55.258501] [error] [63583] [operation_base.cpp:764] Prefill_layer_3 execute fail, error code: 3 

[2025-02-67 08:56:55.373647] [error] [61944] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config 

[2025-02-07 08:56:55.373711] [error] [61944] [comm_pool.h:43] CommPool commCreateFunc fail 

[2025-02-07 08:56:55.373718] [error] [61944] [hccl_runner.cpp:72] AllReduceHcclRunner:0 get hccl comm fail by rank:1 

[2025-02-07 08:56:55.374097] [error] [61944] [hccl_runner.cpp:103] HCCL CommInitClusterInfo ERROR1 should check rankTableFile config