【问题求助】分配NPU空间正常,计算和打印报错: ERR01100 OPS call acl api failed
点赞0收藏0回复0
分享
举报
【问题求助】分配NPU空间正常,计算和打印报错: ERR01100 OPS call acl api failed
已解决
发表于2025-10-21 14:26:07
0 查看
环境信息
(dev) root@gds-ascend:/data# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
(dev) root@gds-ascend:/data# uname -r
5.4.0-216-generic
(dev) root@gds-ascend:/data# python --version
Python 3.10.18
(dev) root@gds-ascend:/data# pip list | grep torch
torch 2.5.1+cpu
torch-npu 2.5.1
(dev) root@gds-ascend:/data# ls
Ascend-cann-kernels-910b_8.2.RC1_linux-x86_64.run Ascend-cann-toolkit_8.2.RC1_linux-x86_64.run Ascend-hdk-910b-npu-firmware_7.7.0.10.220.run miniconda3 models test.py
Ascend-cann-nnal_8.2.RC2_linux-x86_64.run Ascend-hdk-910b-npu-driver_25.2.2_linux-x86-64.run fusion_result.json Miniconda3-latest-Linux-x86_64.sh projects
(dev) root@gds-ascend:/data# npu-smi --version
Command or option is not found.
Usage: npu-smi <Command|-h|-v> [Options...]
npu-smi Command:
-h This command's help information
--help This command's help information
-v Show npu-smi version information
info Show hardware details
set Modify device configuration properties
clear Clear device information
upgrade Upgrade MCU firmware
(dev) root@gds-ascend:/data# npu-smi -v
npu-smi version: 25.2.2
(dev) root@gds-ascend:/data# ls /usr/local/Ascend/
ascend-toolkit/ firmware/ host_servers_setup.sh host_services_setup.sh nnal/
driver/ host_servers_remove.sh host_services_exit.sh host_sys_init.sh version.info
(dev) root@gds-ascend:/data# ls /usr/local/Ascend/ascend-toolkit/
8.2/ 8.2.RC1/ latest/ set_env.sh
(dev) root@gds-ascend:/data# ls /usr/local/Ascend/ascend-toolkit/latest/
acllib/ bisheng_toolkit/ fwkacllib/ lib64/ ops/ python/ test-ops/ var/
atc/ compiler/ hccl/ mindstudio-toolkit/ platform.ini remove_latest_softlink.sh toolkit/ version.cfg
bin/ conf/ include/ opp/ pyACL/ runtime/ tools/ x86_64-linux/
(dev) root@gds-ascend:/data# cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg
# version: 1.0
runtime_running_version=[8.2.0.0.201:8.2.RC1]
compiler_running_version=[8.2.0.0.201:8.2.RC1]
hccl_running_version=[8.2.0.0.201:8.2.RC1]
opp_running_version=[8.2.0.0.201:8.2.RC1]
toolkit_running_version=[8.2.0.0.201:8.2.RC1]
aoe_running_version=[8.2.0.0.201:8.2.RC1]
ncs_running_version=[8.2.0.0.201:8.2.RC1]
runtime_upgrade_version=[8.2.0.0.201:8.2.RC1]
compiler_upgrade_version=[8.2.0.0.201:8.2.RC1]
hccl_upgrade_version=[8.2.0.0.201:8.2.RC1]
opp_upgrade_version=[8.2.0.0.201:8.2.RC1]
toolkit_upgrade_version=[8.2.0.0.201:8.2.RC1]
aoe_upgrade_version=[8.2.0.0.201:8.2.RC1]
ncs_upgrade_version=[8.2.0.0.201:8.2.RC1]
runtime_installed_version=[8.2.0.0.201:8.2.RC1]
compiler_installed_version=[8.2.0.0.201:8.2.RC1]
hccl_installed_version=[8.2.0.0.201:8.2.RC1]
opp_installed_version=[8.2.0.0.201:8.2.RC1]
toolkit_installed_version=[8.2.0.0.201:8.2.RC1]
aoe_installed_version=[8.2.0.0.201:8.2.RC1]
ncs_installed_version=[8.2.0.0.201:8.2.RC1]
(dev) root@gds-ascend:/data#
问题现象如下
(dev) root@gds-ascend:/data# python
Python 3.10.18 (main, Jun 5 2025, 13:14:17) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
/data/miniconda3/envs/dev/lib/python3.10/site-packages/torch_npu/__init__.py:289: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. Do not set it to 1 to prevent some unknown errors
warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \
>>> import torch_npu
>>> x=torch.ones(10).npu()
>>> y=torch.zeros(10).npu()
>>> z=x+y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: call aclnnAdd failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 207848] 2025-10-21-06:14:19.201.275 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
AddAiCore ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-10-21-06:14:19 (PID:207848, Device:0, RankID:-1) ERR01100 OPS call acl api failed
>>> exit()
(dev) root@gds-ascend:/data# echo $LD_LIBRARY_PATH
/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/x86_64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/x86_64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver::/usr/local/Ascend/ascend-toolkit/latest/x86_64-linux/devlib
x,y 变量初始化没问题,npu-smi info 可查看到进程已经分配了显存
root@gds-ascend:/data/projects/LMCache-Ascend# npu-smi info
+------------------------------------------------------------------------------------------------+
| npu-smi 25.2.2 Version: 25.2.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 16 910B4 | OK | 72.1 42 0 / 0 |
| 0 | 0000:15:00.0 | 0 0 / 0 2900 / 32768 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 16 0 | 207848 | python | 116 |
+===========================+===============+====================================================+
环境信息
(dev) root@gds-ascend:/data# cat /etc/os-release NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal (dev) root@gds-ascend:/data# uname -r 5.4.0-216-generic (dev) root@gds-ascend:/data# python --version Python 3.10.18 (dev) root@gds-ascend:/data# pip list | grep torch torch 2.5.1+cpu torch-npu 2.5.1 (dev) root@gds-ascend:/data# ls Ascend-cann-kernels-910b_8.2.RC1_linux-x86_64.run Ascend-cann-toolkit_8.2.RC1_linux-x86_64.run Ascend-hdk-910b-npu-firmware_7.7.0.10.220.run miniconda3 models test.py Ascend-cann-nnal_8.2.RC2_linux-x86_64.run Ascend-hdk-910b-npu-driver_25.2.2_linux-x86-64.run fusion_result.json Miniconda3-latest-Linux-x86_64.sh projects (dev) root@gds-ascend:/data# npu-smi --version Command or option is not found. Usage: npu-smi <Command|-h|-v> [Options...] npu-smi Command: -h This command's help information --help This command's help information -v Show npu-smi version information info Show hardware details set Modify device configuration properties clear Clear device information upgrade Upgrade MCU firmware (dev) root@gds-ascend:/data# npu-smi -v npu-smi version: 25.2.2 (dev) root@gds-ascend:/data# ls /usr/local/Ascend/ ascend-toolkit/ firmware/ host_servers_setup.sh host_services_setup.sh nnal/ driver/ host_servers_remove.sh host_services_exit.sh host_sys_init.sh version.info (dev) root@gds-ascend:/data# ls /usr/local/Ascend/ascend-toolkit/ 8.2/ 8.2.RC1/ latest/ set_env.sh (dev) root@gds-ascend:/data# ls /usr/local/Ascend/ascend-toolkit/latest/ acllib/ bisheng_toolkit/ fwkacllib/ lib64/ ops/ python/ test-ops/ var/ atc/ compiler/ hccl/ mindstudio-toolkit/ platform.ini remove_latest_softlink.sh toolkit/ version.cfg bin/ conf/ include/ opp/ pyACL/ runtime/ tools/ x86_64-linux/ (dev) root@gds-ascend:/data# cat /usr/local/Ascend/ascend-toolkit/latest/version.cfg # version: 1.0 runtime_running_version=[8.2.0.0.201:8.2.RC1] compiler_running_version=[8.2.0.0.201:8.2.RC1] hccl_running_version=[8.2.0.0.201:8.2.RC1] opp_running_version=[8.2.0.0.201:8.2.RC1] toolkit_running_version=[8.2.0.0.201:8.2.RC1] aoe_running_version=[8.2.0.0.201:8.2.RC1] ncs_running_version=[8.2.0.0.201:8.2.RC1] runtime_upgrade_version=[8.2.0.0.201:8.2.RC1] compiler_upgrade_version=[8.2.0.0.201:8.2.RC1] hccl_upgrade_version=[8.2.0.0.201:8.2.RC1] opp_upgrade_version=[8.2.0.0.201:8.2.RC1] toolkit_upgrade_version=[8.2.0.0.201:8.2.RC1] aoe_upgrade_version=[8.2.0.0.201:8.2.RC1] ncs_upgrade_version=[8.2.0.0.201:8.2.RC1] runtime_installed_version=[8.2.0.0.201:8.2.RC1] compiler_installed_version=[8.2.0.0.201:8.2.RC1] hccl_installed_version=[8.2.0.0.201:8.2.RC1] opp_installed_version=[8.2.0.0.201:8.2.RC1] toolkit_installed_version=[8.2.0.0.201:8.2.RC1] aoe_installed_version=[8.2.0.0.201:8.2.RC1] ncs_installed_version=[8.2.0.0.201:8.2.RC1] (dev) root@gds-ascend:/data#问题现象如下
(dev) root@gds-ascend:/data# python Python 3.10.18 (main, Jun 5 2025, 13:14:17) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import torch /data/miniconda3/envs/dev/lib/python3.10/site-packages/torch_npu/__init__.py:289: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. Do not set it to 1 to prevent some unknown errors warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \ >>> import torch_npu >>> x=torch.ones(10).npu() >>> y=torch.zeros(10).npu() >>> z=x+y Traceback (most recent call last): File "<stdin>", line 1, in <module> RuntimeError: call aclnnAdd failed, detail:EZ9999: Inner Error! EZ9999: [PID: 207848] 2025-10-21-06:14:19.201.275 Parse dynamic kernel config fail. TraceBack (most recent call last): AclOpKernelInit failed opType AddAiCore ADD_TO_LAUNCHER_LIST_AICORE failed. [ERROR] 2025-10-21-06:14:19 (PID:207848, Device:0, RankID:-1) ERR01100 OPS call acl api failed >>> exit() (dev) root@gds-ascend:/data# echo $LD_LIBRARY_PATH /usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/x86_64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/lib:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/examples:/usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_1/tests/atbopstest:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/x86_64:/usr/local/Ascend/driver/lib64:/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver::/usr/local/Ascend/ascend-toolkit/latest/x86_64-linux/devlibx,y 变量初始化没问题,
npu-smi info可查看到进程已经分配了显存