环境 altas 800T a2 910b3 npu-smi 25.5.1 ,cann 9.0.0-beta.2
mooncake 是通过目前main分支构建的最新版
我在容器中运行 mooncake_client --global_segment_size=4GB --protocol=ascend --master_server_address="${MASTER_SERVER_ADDRESS}" --metadata_server="${METADATA_SERVER}" --host="${LOCAL_IP}" 的时候报下面错误
查看/root/ascend/log/debug/plog/plog-259_20260430060209546.log
/root/ascend/log/debug/device-1/device-259_20260430055943214.log
但是等一段时间后,device-1又能成功aclrtSetDevice ,这是什么原因造成的?
环境 altas 800T a2 910b3 npu-smi 25.5.1 ,cann 9.0.0-beta.2
mooncake 是通过目前main分支构建的最新版
我在容器中运行 mooncake_client --global_segment_size=4GB --protocol=ascend --master_server_address="${MASTER_SERVER_ADDRESS}" --metadata_server="${METADATA_SERVER}" --host="${LOCAL_IP}" 的时候报下面错误
WARNING: Logging before InitGoogleLogging() is written to STDERR I0430 05:59:38.498884 259 context_manager.cpp:60] ContextManager: found 8 device(s) I0430 05:59:39.120025 259 context_manager.cpp:116] ContextManager: logic 0 -> physical 0 E0430 06:02:09.546612 259 context_manager.cpp:30] Failed to set device, error: 507033, [PID: 259] 2026-04-30-06:02:09.545.377 Inner_Error_Device_Subprocess_Startup_Timeout(E39007): Starting a subprocess on the device timed out. Possible Cause: 1.Failed to start the subprocess. 2. The HDC link is faulty. Solution: Obtain the device log and contact technical support at https://www.hiascend.com/support. TraceBack (most recent call last): tsd client wait response fail, hostpid:186297, receive device response data failed, check hdc service.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:340] Starting a subprocess on the device timed out. TsdOpen failed. devId=1, tdt error=31[FUNC:PrintfTsdError][FILE:runtime.cc][LINE:2618] Check param failed, dev can not be NULL![FUNC:DeviceRetain][FILE:runtime.cc][LINE:3536] Check param failed, dev can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3153] Check param failed, ctx can not be NULL![FUNC:PrimaryContextRetain][FILE:runtime.cc][LINE:3184] Check param failed, context can not be null.[FUNC:SetDevice][FILE:api_impl.cc][LINE:3319] rtSetDevice execution failed, reason=device retain error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65] open device 1 failed, runtime result = 507033.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148] E0430 06:02:09.546815 259 real_client.cpp:541] Failed to initialize ContextManager F0430 06:02:09.546831 259 real_client_main.cpp:112] Failed to setup client: INVALID_PARAMS *** Check failure stack trace: *** command terminated with exit code 250查看/root/ascend/log/debug/plog/plog-259_20260430060209546.log
/root/ascend/log/debug/device-1/device-259_20260430055943214.log
但是等一段时间后,device-1又能成功aclrtSetDevice ,这是什么原因造成的?