MindIE镜像中推理:
cd /urs/local/Ascend/llm_model
bash examples/models/qwen/run_pa.sh -m /data/models/Qwen1.5-7B/weight -c true
报错:
Traceback (most recent call last):
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/Ascend/llm_model/examples/run_pa.py", line 345, in <module>
pa_runner = PARunner(**input_dict)
File "/usr/local/Ascend/llm_model/examples/run_pa.py", line 40, in __init__
self.model = ModelRunner(
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 71, in __init__
self.process_group, self.device = initialize_distributed(self.rank, self.npu_id, world_size)
File "/usr/local/Ascend/llm_model/atb_llm/utils/dist.py", line 93, in initialize_distributed
torch.npu.set_device(device)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 55, in set_device
torch_npu._C._npu_setDevice(device_id)
RuntimeError: GetDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:57 NPU function error: aclrtGetCurrentContext(&used_devices[local_device]), error code is 107002
[ERROR] 2024-11-01-06:53:19 (PID:138986, Device:0, RankID:6) ERR00100 PTA call acl api failed
[Error]: The context is empty.
Check whether acl.rt.set_context or acl.rt.set_device is called.
EL0005: 2024-11-01-06:53:16.445.069 The resources are busy.
Possible Cause: 1. The resources have been occupied. 2. The device is being reset. 3. Software is not ready.
Solution: 1. Close applications not in use. 2. Wait for a while and try again.
TraceBack (most recent call last):
rtGetDeviceCount execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
get device count failed, runtime result = 507899.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
Failed to drvGetPlatformInfo, ret=87[FUNC:DrvGetPlatformInfo][FILE:ai_drv_dev_api.cpp][LINE:79]
halGetDeviceInfo get device type version failed , ret:87[FUNC:Init][FILE:config_manager.cpp][LINE:61]
[Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4916]
The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
MindIE镜像中推理:
cd /urs/local/Ascend/llm_model
bash examples/models/qwen/run_pa.sh -m /data/models/Qwen1.5-7B/weight -c true
报错:
Traceback (most recent call last):
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/python3.10.2/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/Ascend/llm_model/examples/run_pa.py", line 345, in <module>
pa_runner = PARunner(**input_dict)
File "/usr/local/Ascend/llm_model/examples/run_pa.py", line 40, in __init__
self.model = ModelRunner(
File "/usr/local/Ascend/llm_model/atb_llm/runner/model_runner.py", line 71, in __init__
self.process_group, self.device = initialize_distributed(self.rank, self.npu_id, world_size)
File "/usr/local/Ascend/llm_model/atb_llm/utils/dist.py", line 93, in initialize_distributed
torch.npu.set_device(device)
File "/usr/local/python3.10.2/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 55, in set_device
torch_npu._C._npu_setDevice(device_id)
RuntimeError: GetDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:57 NPU function error: aclrtGetCurrentContext(&used_devices[local_device]), error code is 107002
[ERROR] 2024-11-01-06:53:19 (PID:138986, Device:0, RankID:6) ERR00100 PTA call acl api failed
[Error]: The context is empty.
Check whether acl.rt.set_context or acl.rt.set_device is called.
EL0005: 2024-11-01-06:53:16.445.069 The resources are busy.
Possible Cause: 1. The resources have been occupied. 2. The device is being reset. 3. Software is not ready.
Solution: 1. Close applications not in use. 2. Wait for a while and try again.
TraceBack (most recent call last):
rtGetDeviceCount execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
get device count failed, runtime result = 507899.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
Failed to drvGetPlatformInfo, ret=87[FUNC:DrvGetPlatformInfo][FILE:ai_drv_dev_api.cpp][LINE:79]
halGetDeviceInfo get device type version failed , ret:87[FUNC:Init][FILE:config_manager.cpp][LINE:61]
[Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4916]
The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]