昇腾310p卡部署bert训练的分类任务推理服务报错,排查缺失tb_service等服务
收藏回复举报
昇腾310p卡部署bert训练的分类任务推理服务报错,排查缺失tb_service等服务
t('forum.solved') 已解决
新人帖
发表于2026-01-30 18:31:17
0 查看
报错详情:

The Inner error is reported as above. The process exits for this inner error, and the current working operator name is GatherV2.

Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1.

Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging.

[ERROR] 2026-01-29-18:29:00 (PID:67516, Device:0, RankID:-1) ERR00100 PTA call acl api failed.

E90000: [PID: 67516] 2026-01-29-18:26:52.939.377 Compile operator failed, cause: 111Connection refused   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 70, in cann_kb_search_async

    if not public_interact_async("cann_kb_search", kw, key):

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 380, in public_interact_async

    status, msg_mgr = get_msg_obj()

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 322, in get_msg_obj

    return client_get_msg_obj(msg_file_path, file_name)

  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 198, in client_get_msg_obj

    obj = pickle.loads(obj_bytes)

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 942, in RebuildProxy

    return func(token, serializer, incref=incref, **kwds)

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 990, in AutoProxy

    proxy = ProxyType(token, serializer, manager=manager, authkey=authkey,

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 792, in __init__

    self._incref()

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/managers.py", line 846, in _incref

    conn = self._Client(self._token.address, authkey=self._authkey)

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 502, in Client

    c = SocketClient(address)

  File "/home/bml/anaconda3/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient

    s.connect(address)

        TraceBack (most recent call last):

        Fail to get tune knowledge for op[GatherV247, GatherV2].[FUNC:UpdateNodeCompileParams][FILE:op_compiler.cc][LINE:668]

        Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph3[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:119]

        subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:815]

        build graph failed, graph id:46, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]

        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

网上查询的结果:

TBE 算子编译时,无法连接到算子调优知识库(Knowledge Base)服务,核心报错信息为 111Connection refused + Fail to get tune knowledge for op[GatherV2117, GatherV2],后续的 GE 接口调用失败、算子模型构建失败均为该连接问题的连锁反应。

执行ps -ef | grep tb_service,没有发现任何相关进程。未启动知识库相关服务

设置环境变量跳过调优知识库查询

无效,还是报错:

export ASCEND_SKIP_TUNE_KNOWLEDGE=1

export TBE_SKIP_KERNEL_TUNE=1

硬件版本信息

cat /usr/local/Ascend/driver/version.info Version=24.1.0.1 ascendhal_version=7.35.23 aicpu_version=1.0 tdt_version=1.0 log_version=1.0 prof_version=2.0 dvppkernels_version=1.1 tsfw_version=1.0 Innerversion=V100R001C19SPC005B220 compatible_version=[V100R001C13],[V100R001C15],[V100R001C17],[V100R001C18],[V100R001C19] compatible_version_fw=[6.4.0,6.4.99],[7.0.0,7.6.99] package_version=24.1.0.1

本地驱动信息

/tmp/Ascend-cann-kernels-910b_8.0.RC1_linux.run /tmp/Ascend-cann-toolkit_8.0.RC1_linux-x86_64.run

/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-nnal_8.2.RC1_linux-x86_64.run

/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-kernels-310p_8.2.RC1_linux-x86_64.run

/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-cann-toolkit_8.2.RC1_linux-x86_64.run

/home/bml/.storage/mnt/v-gnmqwpg3fs1z5b6t/org/gt/Ascend-mindie_2.0.RC2_linux-x86_64.run

查阅到需要下载

Ascend-cann-toolkit_24.1.0.1_linux-x86_64.run 修复tb相关文件并启动服务,但官网找不到这个包。

希望有大佬指导下如何处理这个问题

我要发帖子