1. 分别起两个docker, 使用 --device=/dev/davinci0,--device=/dev/davinci1 和 -e ASCEND_RT_VISIBLE_DEVICES=0,1 给两个docker分割显卡。
2. 进入docker 后, npu-smi info 能正确显示当前docker分割的显卡设备信息
3. 启动mindie-service出错,始终只能启动 0,1 卡对应的docker , 2,3卡时始终出错, 和启动参数和模型完全无关,更换两个模型后始终也无法正确启动2,3
2,3卡对应的异常日志如下
1. 分别起两个docker, 使用 --device=/dev/davinci0,--device=/dev/davinci1 和 -e ASCEND_RT_VISIBLE_DEVICES=0,1 给两个docker分割显卡。
2. 进入docker 后, npu-smi info 能正确显示当前docker分割的显卡设备信息
3. 启动mindie-service出错,始终只能启动 0,1 卡对应的docker , 2,3卡时始终出错, 和启动参数和模型完全无关,更换两个模型后始终也无法正确启动2,3
2,3卡对应的异常日志如下
[root@a646fe730a47 bin]# grep 'ERROR' ~/mindie/log/ -r /root/mindie/log/debug/mindie-server_114_202509291105923.log:[2025-09-29 11:05:22.294+08:00] [114] [116] [server] [ERROR] [llm_daemon.cpp:90] : [MIE04E010109] [daemon] Process 223 was terminated by signal 9 (Killed) /root/mindie/log/debug/mindie-server_114_202509291105923.log:[2025-09-29 11:05:22.323+08:00] [114] [114] [server] [ERROR] [llm_daemon.cpp:90] : [MIE04E010109] [daemon] Process 221 was terminated by signal 9 (Killed) /root/mindie/log/debug/mindie-batchscheduler_221_20250929110517016.log:[2025-09-29 11:05:22.029+0800] [221] [140322789127872] [batchscheduler] [ERROR] [model.py:61] : [Model] >>> Exception:init:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:292 NPU function error: aclrtGetDeviceCount(&device_count), error code is 207004 /root/mindie/log/debug/mindie-batchscheduler_221_20250929110517016.log:[ERROR] 2025-09-29-11:05:22 (PID:221, Device:-1, RankID:-1) ERR00100 PTA call acl api failed /root/mindie/log/debug/mindie-batchscheduler_221_20250929110517016.log:[ERROR] 2025-09-29-11:05:22 (PID:221, Device:-1, RankID:-1) ERR00100 PTA call acl api failed /root/mindie/log/debug/mindie-batchscheduler_221_20250929110517016.log:[2025-09-29 11:05:22.029+0800] [221] [140322789127872] [batchscheduler] [ERROR] [model.py:64] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'} /root/mindie/log/debug/mindie-batchscheduler_223_20250929110517019.log:[2025-09-29 11:05:22.029+0800] [223] [140068480087744] [batchscheduler] [ERROR] [model.py:61] : [Model] >>> Exception:init:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:292 NPU function error: aclrtGetDeviceCount(&device_count), error code is 207004 /root/mindie/log/debug/mindie-batchscheduler_223_20250929110517019.log:[ERROR] 2025-09-29-11:05:22 (PID:223, Device:-1, RankID:-1) ERR00100 PTA call acl api failed /root/mindie/log/debug/mindie-batchscheduler_223_20250929110517019.log:[ERROR] 2025-09-29-11:05:22 (PID:223, Device:-1, RankID:-1) ERR00100 PTA call acl api failed /root/mindie/log/debug/mindie-batchscheduler_223_20250929110517019.log:[2025-09-29 11:05:22.029+0800] [223] [140068480087744] [batchscheduler] [ERROR] [model.py:64] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'}