使用单张300IDUO卡进行mindie推理不报错，换成两张300IDUO推理就会报这个错。

运行脚本如下：

 

export BIND_CPU=1 

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 

export TP_WORLD_SIZE=4 

export MASTER_PORT=20039 

 

export ATB_LAYER_INTERNAL_TENSOR_REUSE=1 

export ATB_WORKSPACE_MEM_ALLOC_GLOBAL=1 

 

#export BACKEND='hccl' 

export HCCL_OP_BASE_FFTS_MODE_ENABLE=FALSE 

export ATB_OPERATION_EXECUTE_ASYNC=1 

export TASK_QUEUE_ENABLE=1 

export HCCL_BUFFSIZE=110 

export INT8_FORMAT_NZ_ENABLE=1 

export ASDOPS_LOG_LEVEL=INFO 

export ASDOPS_LOG_TO_STDOUT=1 

 

export PYTHONPATH=${llm_path}:$PYTHONPATH 

torchrun --nproc_per_node $TP_WORLD_SIZE --master_port $MASTER_PORT -m examples.run_pa --model_path $1 

报错信息如下：

[2024-08-19 16:02:40.941898] [error] [10517] [hccl_runner.cpp:261] hccl Execute failed, HcclResult:1

[2024-08-19 16:02:40.941945] [error] [10517] [runner.cpp:117] AllReduceHcclRunner_11_0_3_1:2 Execute Failed. st: 12

[2024-08-19 16:02:40.941967] [error] [10517] [graph_runner.cpp:875] LinearRowParallelNoAddRunner_11_0_3:1  node[1] execute fail, runner name:AllReduceHcclRunner

[2024-08-19 16:02:40.941980] [info] [10517] [graph_runner.cpp:381] LinearRowParallelNoAddRunner_11_0_3:1 execute fail

[2024-08-19 16:02:40.941988] [error] [10517] [runner.cpp:117] LinearRowParallelNoAddRunner_11_0_3:2 Execute Failed. st: 12

[2024-08-19 16:02:40.941995] [error] [10517] [graph_runner.cpp:875] AttentionRunner_11_0:1  node[3] execute fail, runner name:LinearRowParallelNoAddRunner

[2024-08-19 16:02:40.942005] [info] [10517] [graph_runner.cpp:381] AttentionRunner_11_0:1 execute fail

[2024-08-19 16:02:40.942012] [error] [10517] [runner.cpp:117] AttentionRunner_11_0:2 Execute Failed. st: 12

[2024-08-19 16:02:40.942018] [error] [10517] [graph_runner.cpp:875] Prefill_layerRunner_11:1  node[0] execute fail, runner name:AttentionRunner

[2024-08-19 16:02:40.942038] [info] [10517] [graph_runner.cpp:381] Prefill_layerRunner_11:1 execute fail

[2024-08-19 16:02:40.942046] [error] [10517] [runner.cpp:117] Prefill_layerRunner_11:2 Execute Failed. st: 12

[2024-08-19 16:02:40.942051] [error] [10517] [operation_base.cpp:666] Prefill_layer_11 execute Prefill_layerRunner_0xaaab6d791fa0 fail

[2024-08-19 16:02:40.942057] [error] [10517] [operation_base.cpp:701] Prefill_layer_11 execute fail, error code: 3

[2024-08-19 16:02:40.942104] [info] [10517] [operation_base.cpp:647] Prefill_layer_12 execute start, workspaceSize:17309696, variantPack:inTensors[0]: dtype: float16, format: nd, shape:[8, 4096], dataSize:65536