4张卡ascend300iPro

mindie-service# ./bin/mindieservice_daemon 跑llama2 ，在1张卡和2张卡，结果都是正确的。但是设置为4张卡，加载模型正确。在模型推理时候发生错误，日志如下：

2024-09-23 06:58:30,097 [INFO] standard_model.py:76 - [Model]   >>> rank-3 enter execute, batch size:1, config:{'EXECUTE_TYPE': '1'}                                                                         

2024-09-23 06:58:30,098 [INFO] standard_model.py:122 - [Model]  >>> Enter generate()                                                                                                                         

2024-09-23 06:58:30,098 [INFO] standard_model.py:186 - [Model]  >>> rank-3 make metadata spend time: 0 batch size: 1                                                                                         

2024-09-23 06:58:30,097 [INFO] standard_model.py:76 - [Model]   >>> rank-1 enter execute, batch size:1, config:{'EXECUTE_TYPE': '1'}                                                                         

2024-09-23 06:58:30,099 [INFO] standard_model.py:122 - [Model]  >>> Enter generate()                                                                                                                         

2024-09-23 06:58:30,099 [INFO] standard_model.py:186 - [Model]  >>> rank-1 make metadata spend time: 0 batch size: 1                                                                                         

2024-09-23 06:58:30,098 [INFO] standard_model.py:76 - [Model]   >>> rank-2 enter execute, batch size:1, config:{'EXECUTE_TYPE': '1'}                                                                         

2024-09-23 06:58:30,100 [INFO] standard_model.py:122 - [Model]  >>> Enter generate()                                                                                                                         

2024-09-23 06:58:30,098 [INFO] standard_model.py:76 - [Model]   >>> rank-0 enter execute, batch size:1, config:{'EXECUTE_TYPE': '1'}                                                                         

2024-09-23 06:58:30,101 [INFO] standard_model.py:122 - [Model]  >>> Enter generate()                                                                                                                         

2024-09-23 06:58:30,101 [INFO] standard_model.py:186 - [Model]  >>> rank-2 make metadata spend time: 0 batch size: 1                                                                                         

2024-09-23 06:58:30,102 [INFO] standard_model.py:186 - [Model]  >>> rank-0 make metadata spend time: 1 batch size: 1                                                                                         

[2024-09-23 06:58:30.102227] [fatal] [11344] [12440] [operation_torch.cpp:202]TransdataOperation executeCount:2, statistic:[totalTime:0, createTensorTime:14, planSetupTime:520, planAsyncTime:0, planExecut 

eTime:1457734, streamSyncTime:0, tillingCopyTime:0, getBestKernelTime:0, kernelExecuteTime:0, kernelCacheHitCount:0, kernelCacheMissCount:0, mallocTorchTensorSize:0]                                        

[2024-09-23 06:58:30.103343] [fatal] [11342] [12666] [operation_torch.cpp:202]TransdataOperation executeCount:2, statistic:[totalTime:1, createTensorTime:17, planSetupTime:478, planAsyncTime:0, planExecut 

eTime:1457404, streamSyncTime:0, tillingCopyTime:0, getBestKernelTime:0, kernelExecuteTime:0, kernelCacheHitCount:0, kernelCacheMissCount:0, mallocTorchTensorSize:0]                                        

[2024-09-23 06:58:30.105196] [fatal] [11343] [12858] [operation_torch.cpp:202]TransdataOperation executeCount:2, statistic:[totalTime:0, createTensorTime:12, planSetupTime:441, planAsyncTime:0, planExecut 

eTime:1479271, streamSyncTime:0, tillingCopyTime:0, getBestKernelTime:0, kernelExecuteTime:0, kernelCacheHitCount:0, kernelCacheMissCount:0, mallocTorchTensorSize:0]                                        

[2024-09-23 06:58:30.108844] [fatal] [11341] [12668] [operation_torch.cpp:202]TransdataOperation executeCount:2, statistic:[totalTime:1, createTensorTime:12, planSetupTime:433, planAsyncTime:0, planExecut 

eTime:1480624, streamSyncTime:0, tillingCopyTime:0, getBestKernelTime:0, kernelExecuteTime:0, kernelCacheHitCount:0, kernelCacheMissCount:0, mallocTorchTensorSize:0]                                        

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.464 [transport.cc:238][12858]errNo[0x0000000005000002] ptr [pimpl_] is nullptr, return HCCL_E_PTR                                

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.488 [all_reduce_doubling.cc:93][12858]call trace: hcclRet -> 2                                                                   

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.496 [all_reduce_doubling.cc:42][12858]call trace: hcclRet -> 2                                                                   

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.504 [coll_executor_base.cc:33][12858][CollExecutorBase][RunTemplate]run executor rank[2] rank size[4] failed                     

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.511 [coll_all_reduce_for_310p_doubling_executor.cc:92][12858]call trace: hcclRet -> 2                                            

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.523 [coll_all_reduce_executor.cc:235][12858][CollAllReduceExecutor][RunLoop]errNo[0x0000000005000002]kernel run error, tag[AllRe 

duce_10.244.0.1%cni0_60000_0_1727074680796169], inputMem ptr[0x12c500000000], outputMem ptr[0x12c507800000], count[32768], dataType[3], reduce op type[0]                                                    

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.531 [coll_all_reduce_executor.cc:174][12858]call trace: hcclRet -> 2                                                             

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.538 [coll_all_reduce_executor.cc:61][12858][CollAllReduceExecutor][Orchestrate]errNo[0x0000000005000002]all reudce excutor kerne 

l run failed 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.546 [hccl_communicator.cc:3368][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.556 [hccl_communicator.cc:2768][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.564 [hccl_comm.cc:298][12858]call trace: hcclRet -> 2 

[INFO] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.571 [op_base.cc:695][12858][HCCL_TRACE]HcclAllReduce identifier[AllReduce_10.244.0.1%cni0_60000_0_1727074680796169] 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.579 [op_base.cc:695][12858]call trace: hcclRet -> 2 

[2024-09-23 06:58:30.111672] [error] [11343] [12858] [model.cpp:412]execute node[2] fail, error code: 3 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.918 [transport.cc:238][12858]errNo[0x0000000005000002] ptr [pimpl_] is nullptr, return HCCL_E_PTR 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.928 [all_reduce_doubling.cc:93][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.934 [all_reduce_doubling.cc:42][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.941 [coll_executor_base.cc:33][12858][CollExecutorBase][RunTemplate]run executor rank[2] rank size[4] failed 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.948 [coll_all_reduce_for_310p_doubling_executor.cc:92][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.958 [coll_all_reduce_executor.cc:235][12858][CollAllReduceExecutor][RunLoop]errNo[0x0000000005000002]kernel run error, tag[AllRe 

duce_10.244.0.1%cni0_60000_0_1727074680796169], inputMem ptr[0x12c500000000], outputMem ptr[0x12c507800000], count[32768], dataType[3], reduce op type[0] 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.966 [coll_all_reduce_executor.cc:174][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.974 [coll_all_reduce_executor.cc:61][12858][CollAllReduceExecutor][Orchestrate]errNo[0x0000000005000002]all reudce excutor kerne 

l run failed 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.981 [hccl_communicator.cc:3368][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.988 [hccl_communicator.cc:2768][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.995 [hccl_comm.cc:298][12858]call trace: hcclRet -> 2 

[INFO] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.571 [op_base.cc:695][12858][HCCL_TRACE]HcclAllReduce identifier[AllReduce_10.244.0.1%cni0_60000_0_1727074680796169]               

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.579 [op_base.cc:695][12858]call trace: hcclRet -> 2                                                                              

[2024-09-23 06:58:30.111672] [error] [11343] [12858] [model.cpp:412]execute node[2] fail, error code: 3                                                                                                      

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.918 [transport.cc:238][12858]errNo[0x0000000005000002] ptr [pimpl_] is nullptr, return HCCL_E_PTR                                

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.928 [all_reduce_doubling.cc:93][12858]call trace: hcclRet -> 2                                                                   

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.934 [all_reduce_doubling.cc:42][12858]call trace: hcclRet -> 2                                                                   

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.941 [coll_executor_base.cc:33][12858][CollExecutorBase][RunTemplate]run executor rank[2] rank size[4] failed                     

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.948 [coll_all_reduce_for_310p_doubling_executor.cc:92][12858]call trace: hcclRet -> 2                                            

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.958 [coll_all_reduce_executor.cc:235][12858][CollAllReduceExecutor][RunLoop]errNo[0x0000000005000002]kernel run error, tag[AllRe 

duce_10.244.0.1%cni0_60000_0_1727074680796169], inputMem ptr[0x12c500000000], outputMem ptr[0x12c507800000], count[32768], dataType[3], reduce op type[0]                                                    

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.966 [coll_all_reduce_executor.cc:174][12858]call trace: hcclRet -> 2                                                             

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.974 [coll_all_reduce_executor.cc:61][12858][CollAllReduceExecutor][Orchestrate]errNo[0x0000000005000002]all reudce excutor kerne 

l run failed

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.981 [hccl_communicator.cc:3368][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.988 [hccl_communicator.cc:2768][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.111.995 [hccl_comm.cc:298][12858]call trace: hcclRet -> 2 

[INFO] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.001 [op_base.cc:695][12858][HCCL_TRACE]HcclAllReduce identifier[AllReduce_10.244.0.1%cni0_60000_0_1727074680796169] 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.008 [op_base.cc:695][12858]call trace: hcclRet -> 2 

[2024-09-23 06:58:30.112032] [error] [11343] [12858] [model.cpp:412]execute node[3] fail, error code: 3 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.220 [transport.cc:238][12858]errNo[0x0000000005000002] ptr [pimpl_] is nullptr, return HCCL_E_PTR 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.229 [all_reduce_doubling.cc:93][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.235 [all_reduce_doubling.cc:42][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.242 [coll_executor_base.cc:33][12858][CollExecutorBase][RunTemplate]run executor rank[2] rank size[4] failed 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.249 [coll_all_reduce_for_310p_doubling_executor.cc:92][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.260 [coll_all_reduce_executor.cc:235][12858][CollAllReduceExecutor][RunLoop]errNo[0x0000000005000002]kernel run error, tag[AllRe 

duce_10.244.0.1%cni0_60000_0_1727074680796169], inputMem ptr[0x12c500000000], outputMem ptr[0x12c507800000], count[32768], dataType[3], reduce op type[0] 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.268 [coll_all_reduce_executor.cc:174][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.276 [coll_all_reduce_executor.cc:61][12858][CollAllReduceExecutor][Orchestrate]errNo[0x0000000005000002]all reudce excutor kern$ 

l run failed 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.283 [hccl_communicator.cc:3368][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.289 [hccl_communicator.cc:2768][12858]call trace: hcclRet -> 2 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.296 [hccl_comm.cc:298][12858]call trace: hcclRet -> 2 

[INFO] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.303 [op_base.cc:695][12858][HCCL_TRACE]HcclAllReduce identifier[AllReduce_10.244.0.1%cni0_60000_0_1727074680796169] 

[ERROR] HCCL(11343,mindieservice_backend_connector):2024-09-23-06:58:30.112.318 [op_base.cc:695][12858]call trace: hcclRet -> 2 

[2024-09-23 06:58:30.112342] [error] [11343] [12858] [model.cpp:412]execute node[4] fail, error code: 3

--- Logging error --- 

Traceback (most recent call last): 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 125, in __generate 

    request_tokens_np, eof_np = self.__handle_requests(requests, is_prefill) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 190, in __handle_requests 

    request_tokens_np, eos_np, stop_generate = self.generator.generate_token(metadata) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 66, in generate_token 

    tokens, eos_np, stop_generate = self.plugin.generate_token(input_metadata) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/plugins/plugin.py", line 12, in generate_token 

    logits = self.generator_backend.forward(model_inputs) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 52, in forward 

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 60, in forward 

    logits = self.forward_tensor( 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 86, in forward_tensor 

    logits = self.model_runner.forward( 

  File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 100, in forward 

    return self.model.forward(**kwargs) 

  File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 229, in forward 

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) 

  File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 200, in execute_ascend_operator 

    acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param) 

RuntimeError: execute\u5931\u8d25 \u8be6\u7ec6\u4fe1\u606f\u89c1Ascend\u5b98\u65b9\u6587\u6863\uff0c\u8bf7\u5f00\u542f\u65e5\u5fd7\u8fdb\u4e00\u6b65\u5b9a\u4f4d\u95ee\u9898\u3002

 

During handling of the above exception, another exception occurred: 

 

Traceback (most recent call last): 

  File "/usr/local/python3.10.2/lib/python3.10/logging/__init__.py", line 1103, in emit 

    stream.write(msg + self.terminator) 

UnicodeEncodeError: 'ascii' codec can't encode characters in position 85-86: ordinal not in range(128)  

Call stack: 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/model.py", line 41, in execute 

    return self.python_model.execute(requests, config_map) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 81, in execute 

    return self._prefill(requests) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 95, in _prefill 

    return self.__generate(requests, is_prefill=True) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 137, in __generate 

    logger.exception(f"[Model]\t>>> Exception:{e}")  

Message: '[Model]\t>>> Exception:execute\u5931\u8d25 \u8be6\u7ec6\u4fe1\u606f\u89c1Ascend\u5b98\u65b9\u6587\u6863\uff0c\u8bf7\u5f00\u542f\u65e5\u5fd7\u8fdb\u4e00\u6b65\u5b9a\u4f4d\u95ee\u9898\u3002\n' 

Arguments: () 

2024-09-23 06:58:30,111 [ERROR] standard_model.py:137 - [Model] >>> Exception:execute\u5931\u8d25 \u8be6\u7ec6\u4fe1\u606f\u89c1Ascend\u5b98\u65b9\u6587\u6863\uff0c\u8bf7\u5f00\u542f\u65e5\u5fd7\u8fdb\u4e 

00\u6b65\u5b9a\u4f4d\u95ee\u9898\u3002 

Traceback (most recent call last): 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 125, in __generate 

    request_tokens_np, eof_np = self.__handle_requests(requests, is_prefill) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/model_wrapper/standard_model.py", line 190, in __handle_requests 

    request_tokens_np, eos_np, stop_generate = self.generator.generate_token(metadata) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/generator.py", line 66, in generate_token 

    tokens, eos_np, stop_generate = self.plugin.generate_token(input_metadata) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/plugins/plugin.py", line 12, in generate_token 

    logits = self.generator_backend.forward(model_inputs) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 52, in forward 

    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 60, in forward 

    logits = self.forward_tensor( 

  File "/usr/local/python3.10.2/lib/python3.10/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 86, in forward_tensor 

    logits = self.model_runner.forward( 

  File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 100, in forward 

    return self.model.forward(**kwargs) 

  File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 229, in forward 

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) 

  File "/usr/local/Ascend/llm_model/atb_llm/models/base/flash_causal_lm.py", line 200, in execute_ascend_operator 

    acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param)