aclrtSetDevice时报 Inner_Error_Device_Subprocess_Startup_Timeout(E39007): Starting a subprocess on the device timed out.
收藏回复举报
aclrtSetDevice时报 Inner_Error_Device_Subprocess_Startup_Timeout(E39007): Starting a subprocess on the device timed out.
t('forum.solved') 已解决
发表于2026-04-30 14:08:39
0 查看

环境 altas 800T a2 910b3  npu-smi 25.5.1 ,cann 9.0.0-beta.2

mooncake 是通过目前main分支构建的最新版

我在容器中运行 mooncake_client --global_segment_size=4GB --protocol=ascend --master_server_address="${MASTER_SERVER_ADDRESS}" --metadata_server="${METADATA_SERVER}" --host="${LOCAL_IP}" 的时候报下面错误

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0430 05:59:38.498884   259 context_manager.cpp:60] ContextManager: found 8 device(s)
I0430 05:59:39.120025   259 context_manager.cpp:116] ContextManager: logic 0 -> physical 0
E0430 06:02:09.546612   259 context_manager.cpp:30] Failed to set device, error: 507033, [PID: 259] 2026-04-30-06:02:09.545.377 Inner_Error_Device_Subprocess_Startup_Timeout(E39007): Starting a subprocess on the device timed out.
        Possible Cause: 1.Failed to start the subprocess. 2. The HDC link is faulty.
        Solution: Obtain the device log and contact technical support at https://www.hiascend.com/support.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:186297, receive device response data failed, check hdc service.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:340]
        Starting a subprocess on the device timed out.
        TsdOpen failed. devId=1, tdt error=31[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618]
        Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536]
        Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153]
        Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184]
        Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3319]
        rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
        open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
E0430 06:02:09.546815   259 real_client.cpp:541] Failed to initialize ContextManager
F0430 06:02:09.546831   259 real_client_main.cpp:112] Failed to setup client: INVALID_PARAMS
*** Check failure stack trace: ***
command terminated with exit code 250

查看/root/ascend/log/debug/plog/plog-259_20260430060209546.log 

[ERROR] TDT(259,mooncake_client):2026-04-30-06:02:09.537.612 [hdc_client.cpp:161][TsdRecvData][tid:259] [deviceId=1][sessionId=1] recv msg failed
[ERROR] TDT(259,mooncake_client):2026-04-30-06:02:09.537.640 [process_mode_manager.cpp:339][WaitRsp][tid:259] tsd client wait response fail, hostpid:186297, receive device response data failed, check hdc service.
[ERROR] TDT(259,mooncake_client):2026-04-30-06:02:09.545.400 [process_mode_manager.cpp:133][OpenProcess][tid:259] Wait open response from device failed.
[ERROR] TDT(259,mooncake_client):2026-04-30-06:02:09.545.409 [tsd_client.cpp:45][TsdOpenEx][tid:259] TsdOpenEx failed, deviceId[1].
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.545.498 [runtime.cc:2618]259 PrintfTsdError:TsdOpen failed. devId=1, tdt error=31
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.545.517 [runtime.cc:3481]259 DeviceRetain:Start aicpu executor failed, retCode=0x7020009, devId=1
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.054 [runtime.cc:3536]259 DeviceRetain:Check param failed, dev can not be NULL!
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.071 [runtime.cc:3153]259 PrimaryContextRetain:Check param failed, dev can not be NULL!
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.081 [runtime.cc:3184]259 PrimaryContextRetain:Check param failed, ctx can not be NULL!
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.091 [api_impl.cc:3319]259 SetDevice:Check param failed, context can not be null.
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.102 [api_error.cc:2548]259 SetDevice:Set device failed, device_id=1, the current input soc version is null.
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.122 [api_c_device.cc:91]259 rtSetDevice:ErrCode=507033, desc=[device retain error], InnerCode=0x7010006
[ERROR] RUNTIME(259,mooncake_client):2026-04-30-06:02:09.546.129 [error_message_manage.cc:65]259 FuncErrorReason:rtSetDevice execution failed, reason=device retain error
[ERROR] ASCENDCL(259,mooncake_client):2026-04-30-06:02:09.546.567 [device.cpp:53]259 aclrtSetDeviceImpl:open device 1 failed, runtime result = 507033.

/root/ascend/log/debug/device-1/device-259_20260430055943214.log

[ERROR] DRV(1492,aicpu_scheduler):2026-04-30-13:59:38.872.877 [drv_share_log.c:238][event-sche] [share_log_read_in_single_module] Not sched cpu. (chip_id=0; cpu_num=8; cur_processor_id=1)

但是等一段时间后,device-1又能成功aclrtSetDevice ,这是什么原因造成的?

我要发帖子