[2024-09-04 11:50:29.888408] [error] [45740] [all_reduce_hccl_runner.cpp:54] hccl Execute failed, HcclResult:2
[2024-09-04 11:50:29.888426] [error] [45740] [runner.cpp:117] AllReduceHcclRunner_29_0_4_1:3 Execute Failed. st: 12
[2024-09-04 11:50:29.888436] [error] [45740] [graph_runner.cpp:877] LinearRowParallelNoAddRunner_29_0_4:2  node[1] execute fail, runner name:AllReduceHcclRunner
[2024-09-04 11:50:29.888445] [error] [45740] [runner.cpp:117] LinearRowParallelNoAddRunner_29_0_4:3 Execute Failed. st: 12
[2024-09-04 11:50:29.888452] [error] [45740] [graph_runner.cpp:877] AttentionRunner_29_0:2  node[4] execute fail, runner name:LinearRowParallelNoAddRunner
[2024-09-04 11:50:29.888461] [error] [45740] [runner.cpp:117] AttentionRunner_29_0:3 Execute Failed. st: 12
[2024-09-04 11:50:29.888467] [error] [45740] [graph_runner.cpp:877] Prefill_layerRunner_29:2  node[0] execute fail, runner name:AttentionRunner
[2024-09-04 11:50:29.888476] [error] [45740] [runner.cpp:117] Prefill_layerRunner_29:3 Execute Failed. st: 12
[2024-09-04 11:50:29.888483] [error] [45740] [operation_base.cpp:679] Prefill_layer_29 execute Prefill_layerRunner fail
[2024-09-04 11:50:29.888489] [error] [45740] [operation_base.cpp:719] Prefill_layer_29 execute fail, error code: 3
[2024-09-04 11:50:29.888496] [error] [43611] [45740] [model.cpp:412]execute node[28] fail, error code: 3
[2024-09-04 11:50:29.889091] [error] [45740] [all_reduce_hccl_runner.cpp:54] hccl Execute failed, HcclResult:2
[2024-09-04 11:50:29.889109] [error] [45740] [runner.cpp:117] AllReduceHcclRunner_30_0_4_1:3 Execute Failed. st: 12
[2024-09-04 11:50:29.889120] [error] [45740] [graph_runner.cpp:877] LinearRowParallelNoAddRunner_30_0_4:2  node[1] execute fail, runner name:AllReduceHcclRunner
[2024-09-04 11:50:29.889130] [error] [45740] [runner.cpp:117] LinearRowParallelNoAddRunner_30_0_4:3 Execute Failed. st: 12
[2024-09-04 11:50:29.889137] [error] [45740] [graph_runner.cpp:877] AttentionRunner_30_0:2  node[4] execute fail, runner name:LinearRowParallelNoAddRunner
[2024-09-04 11:50:29.889145] [error] [45740] [runner.cpp:117] AttentionRunner_30_0:3 Execute Failed. st: 12
[2024-09-04 11:50:29.889152] [error] [45740] [graph_runner.cpp:877] Prefill_layerRunner_30:2  node[0] execute fail, runner name:AttentionRunner
[2024-09-04 11:50:29.889160] [error] [45740] [runner.cpp:117] Prefill_layerRunner_30:3 Execute Failed. st: 12
[2024-09-04 11:50:29.889167] [error] [45740] [operation_base.cpp:679] Prefill_layer_30 execute Prefill_layerRunner fail
[2024-09-04 11:50:29.889173] [error] [45740] [operation_base.cpp:719] Prefill_layer_30 execute fail, error code: 3
[2024-09-04 11:50:29.889180] [error] [43611] [45740] [model.cpp:412]execute node[29] fail, error code: 3