报错详情:
The Inner error is reported as above. The process exits for this inner error, and the current working operator name is GatherV2.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[ERROR] 2026-01-29-18:29:00 (PID:67516, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
E90000: [PID: 67516] 2026-01-29-18:26:52.939.377 Compile operator failed, cause: 111Connection refused File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 70, in cann_kb_search_async
if not public_interact_async("cann_kb_search", kw, key):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 380, in public_interact_async
status, msg_mgr = get_msg_obj()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 322, in get_msg_obj
return client_get_msg_obj(msg_file_path, file_name)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 198, in client_get_msg_obj
obj = pickle.loads(obj_bytes)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 942, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 990, in AutoProxy
proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 792, in __init__
self._incref()
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 846, in _incref
conn = self._Client(self._token.address, authkey=self._authkey)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 502, in Client
c = SocketClient(address)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
TraceBack (most recent call last):
Fail to get tune knowledge for op[GatherV247, GatherV2].[FUNC:UpdateNodeCompileParams][FILE:op_compiler.cc][LINE:668]
Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph3[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:815]
build graph failed, graph id:46, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
网上查询的结果:
TBE 算子编译时,无法连接到算子调优知识库(Knowledge Base)服务,核心报错信息为 111Connection refused + Fail to get tune knowledge for op[GatherV2117, GatherV2],后续的 GE 接口调用失败、算子模型构建失败均为该连接问题的连锁反应。
执行ps -ef | grep tb_service,没有发现任何相关进程。未启动知识库相关服务
设置环境变量跳过调优知识库查询
无效,还是报错:
export ASCEND_SKIP_TUNE_KNOWLEDGE=1
export TBE_SKIP_KERNEL_TUNE=1
硬件版本信息
cat /usr/local/Ascend/driver/version.info Version=24.1.0.1 ascendhal_version=7.35.23 aicpu_version=1.0 tdt_version=1.0 log_version=1.0 prof_version=2.0 dvppkernels_version=1.1 tsfw_version=1.0 Innerversion=V100R001C19SPC005B220 compatible_version=[V100R001C13],[V100R001C15],[V100R001C17],[V100R001C18],[V100R001C19] compatible_version_fw=[6.4.0,6.4.99],[7.0.0,7.6.99] package_version=24.1.0.1
本地驱动信息
/tmp/Ascend-cann-kernels-910b_8.0.RC1_linux.run /tmp/Ascend-cann-toolkit_8.0.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-nnal_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-kernels-310p_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-toolkit_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-mindie_2.0.RC2_linux-x86_64.run
查阅到需要下载
Ascend-cann-toolkit_24.1.0.1_linux-x86_64.run 修复tb相关文件并启动服务,但官网找不到这个包。
希望有大佬指导下如何处理这个问题
The Inner error is reported as above. The process exits for this inner error, and the current working operator name is GatherV2.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.
Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.
[ERROR] 2026-01-29-18:29:00 (PID:67516, Device:0, RankID:-1) ERR00100 PTA call acl api failed.
E90000: [PID: 67516] 2026-01-29-18:26:52.939.377 Compile operator failed, cause: 111Connection refused File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 70, in cann_kb_search_async
if not public_interact_async("cann_kb_search", kw, key):
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 380, in public_interact_async
status, msg_mgr = get_msg_obj()
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 322, in get_msg_obj
return client_get_msg_obj(msg_file_path, file_name)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 198, in client_get_msg_obj
obj = pickle.loads(obj_bytes)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 942, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 990, in AutoProxy
proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 792, in __init__
self._incref()
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 846, in _incref
conn = self._Client(self._token.address, authkey=self._authkey)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 502, in Client
c = SocketClient(address)
File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
s.connect(address)
TraceBack (most recent call last):
Fail to get tune knowledge for op[GatherV247, GatherV2].[FUNC:UpdateNodeCompileParams][FILE:op_compiler.cc][LINE:668]
Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph3[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119]
subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:815]
build graph failed, graph id:46, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
网上查询的结果:
TBE 算子编译时,无法连接到算子调优知识库(Knowledge Base)服务,核心报错信息为 111Connection refused + Fail to get tune knowledge for op[GatherV2117, GatherV2],后续的 GE 接口调用失败、算子模型构建失败均为该连接问题的连锁反应。
执行ps -ef | grep tb_service,没有发现任何相关进程。未启动知识库相关服务
设置环境变量跳过调优知识库查询
无效,还是报错:
export ASCEND_SKIP_TUNE_KNOWLEDGE=1
export TBE_SKIP_KERNEL_TUNE=1
硬件版本信息
cat /usr/local/Ascend/driver/version.info Version=24.1.0.1 ascendhal_version=7.35.23 aicpu_version=1.0 tdt_version=1.0 log_version=1.0 prof_version=2.0 dvppkernels_version=1.1 tsfw_version=1.0 Innerversion=V100R001C19SPC005B220 compatible_version=[V100R001C13],[V100R001C15],[V100R001C17],[V100R001C18],[V100R001C19] compatible_version_fw=[6.4.0,6.4.99],[7.0.0,7.6.99] package_version=24.1.0.1
本地驱动信息
/tmp/Ascend-cann-kernels-910b_8.0.RC1_linux.run /tmp/Ascend-cann-toolkit_8.0.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-nnal_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-kernels-310p_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-toolkit_8.2.RC1_linux-x86_64.run
/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-mindie_2.0.RC2_linux-x86_64.run
查阅到需要下载
Ascend-cann-toolkit_24.1.0.1_linux-x86_64.run 修复tb相关文件并启动服务,但官网找不到这个包。
希望有大佬指导下如何处理这个问题