【问题现象】

mindie2.0.T3镜像版本，进行qwen2.5-0.5B模型单卡推理，推理服务化拉起失败，报错："error":"Failed to get engine response.","error_type":"Incomplete Generation"

【产品信息】

MindIE镜像：2.0.T3.1-800I-A2-py311-openeuler24.03-lts

模型：Qwen2.5-0.5B-Instruct

【分析过程】

1、执行如下命令进行Qwen2.5-0.5B-Instruct纯模型推理，其中${weight_path}表示模型权重路径

cd /usr/local/Ascend/atb-models/examples/models/qwen

bash run_pa.sh --model-path ${weight_path}

报错信息：

[2025-03-31 10:35:45,723] [9330] [281457119611168] [llm] [INFO] [logging.py-331] : Can not get device info, you can use BIND_CPU=0 to skip.

[2025-03-31 10:35:46,283] [9330] [281457119611168] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 509, in <module>

    pa_runner = PARunner(**input_dict)

                ^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 74, in __init__

    self.model = ModelRunner(

                 ^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 142, in __init__

    self.process_group, self.device = initialize_distributed(self.rank, self.npu_id, world_size)

                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/Ascend/atb-models/atb_llm/utils/dist.py", line 80, in initialize_distributed

    torch.npu.set_device(device)

  File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 58, in set_device

    torch_npu._C._npu_setDevice(device_id)

RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:226 NPU function error: c10_npu::SetDevice(device_id_), error code is 107001

[ERROR] 2025-03-31-10:35:46 (PID:9336, Device:0, RankID:6) ERR00100 PTA call acl api failed

[Error]: Invalid device ID.

        Check whether the device ID is valid.

EE1001: [PID: 9336] 2025-03-31-10:35:46.508.981 The argument is invalid.Reason: Set visible device failed, invalid device=6, input visible devices:0,1,2,3,4,5,6,7

        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.

        TraceBack (most recent call last):

        rtSetDevice execute failed, reason=[device id error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

        open device 6 failed, runtime result = 107001.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

2、单卡推理时，执行命令：export ASCEND_RT_VISIBLE_DEVICES=0，指定运行在id=0的逻辑卡上

3、重新运行推理，仍旧报错，报错信息如下：

[2025-03-31 11:55:22.679578] [error] [19353] [runner.cpp:115] PluginRunner_55_0_0:1 Execute Failed. st: 12

[2025-03-31 11:55:22.679586] [error] [19353] [graph_runner.cpp:906] LinearNoQuantRunner_55_0:0  node[0] execute fail, runner name:PluginRunner

[2025-03-31 11:55:22.679594] [info] [19353] [graph_runner.cpp:398] LinearNoQuantRunner_55_0:0 execute fail

[2025-03-31 11:55:22.679600] [debug] [19353] [probe.cpp:47] executeCount:0

[2025-03-31 11:55:22.679605] [debug] [19353] [probe.cpp:47] executeCount:0

[2025-03-31 11:55:22.679610] [error] [19353] [runner.cpp:115] LinearNoQuantRunner_55_0:1 Execute Failed. st: 12

[2025-03-31 11:55:22.679617] [error] [19353] [graph_runner.cpp:906] LmHeadRunner_55:0  node[0] execute fail, runner name:LinearNoQuantRunner

[2025-03-31 11:55:22.679623] [info] [19353] [graph_runner.cpp:398] LmHeadRunner_55:0 execute fail

4、针对使用的mindie是2.0.0.T3的镜像，Qwen2系列模型可能开启了MATMUL格式转换和EACLNN接口调用，因此单卡跑Qwen系列模型时，需要关闭这两个参数：

export MATMUL_ND_NZ_ENABLE=0

export ENABLE_ACLNN_MATMUL_BACKEND=0

【故障根因】

mindie2.00.T3镜像中，Qwen2系列模型开启MATMUL_ND_NZ_ENABLE和ENABLE_ACLNN_MATMUL_BACKEND

【解决方法】

关闭MATMUL_ND_NZ_ENABLE和ENABLE_ACLNN_MATMUL_BACKEND，并在每次进入容器时，都设置一次

【其他】

mindie 1.0.0镜像、mindie 2.0.T6及之后的镜像不涉及，可直接使用