容器共享内存不足导致启动mindie service服务报错:LLMInferEngine failed to init LLMInferModels
收藏回复举报
容器共享内存不足导致启动mindie service服务报错:LLMInferEngine failed to init LLMInferModels
发表于2025-04-07 09:42:38
0 查看

【问题现象】

执行如下命令启动MindIE服务报错LLMInferEngine failed to init LLMInferModels

cd /usr/local/Ascend/mindie/latest/mindie-service; ./bin/mindieservice_daemon

【定位过程】
1)   直接查看日志/usr/local/Ascend/mindie/latest/mindie-service/logs/mindie-server.log,发现内容为空

2)设置日志级别为debug并打屏 
export MINDIE_LOG_LEVEL=debug; export MINDIE_LOG_TO_STDOUT=1
可以看到报错关键信息当前容器中的shared memory size是64M,不足以启动服务。

[2025-03-10 20:26:36.599+08:00] [324] [325] [mindie-server] [INFO] [connector_launcher.cpp:127] : [model_backend] Signal SIGCHLD handler has been registered.
[2025-03-10 20:26:36.599+08:00] [324] [325] [mindie-server] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:64MB, and available shared memory size:63.625MB.
[2025-03-10 20:26:36.599+08:00] [324] [325] [mindie-server] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-03-10 20:26:36.604+08:00] [324] [325] [mindie-server] [ERROR] [share_memory.cpp:172] : [MIE04E120407] [model_backend] Shared memory available is not enough on the filesystem.

[2025-03-10 20:26:36.604+08:00] [324] [325] [mindie-server] [ERROR] [share_memory.cpp:27] : [MIE04E120407] [model_backend] Available shared memory size is not enough.
[2025-03-10 20:26:36.604+08:00] [324] [325] [mindie-server] [ERROR] [master_IPC_communicator.cpp:156] : [MIE04E120414] [model_backend] Failed to create response shared memory
[2025-03-10 20:26:36.604+08:00] [324] [325] [mindie-server] [ERROR] [connector_launcher.cpp:148] : [MIE04E120414] [model_backend] Failed to setup message channel.
[2025-03-10 20:26:36.605+08:00] [324] [325] [mindie-server] [ERROR] [model_backend.cpp:158] : [MIE04E12041C] [model_backend] Failed to launcher connector agent.
[2025-03-10 20:26:36.605+08:00] [324] [A800T-A2-1] [system] [model_backend] [init model backend] [fail]
[2025-03-10 20:26:36.605+08:00] [324] [325] [mindie-server] [DEBUG] [connector_launcher.cpp:185] : [model_backend] Ready to stop connector processes.
[2025-03-10 20:26:36.605345] [324] [281460141191104] [llm] [ERROR] [llm_infer_model_instance.cpp:47] [MIE05E030001] Initialize modelBackends_ failed, export MINDIE_LLM_PYTHON_LOG_TO_STDOUT=1 export MINDIE_LLM_PYTHON_LOG_TO_FILE=1 to find error in pythonlog
[2025-03-10 20:26:36.605381] [324] [281460141191104] [llm] [ERROR] [llm_infer_model.cpp:25] Init model instance failed.
LLMInferEngine failed to init LLMInferModels
[2025-03-10 20:26:36.605+08:00] [324] [A800T-A2-1] [system] [server] [start engine(mindieservice_llm_engine)] [fail]
[2025-03-10 20:26:36.605+08:00] [324] [325] [mindie-server] [ERROR] [infer_backend_manager.cpp:236] : [MIE04E06011C] [infer_backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-03-10 20:26:36.605+08:00] [324] [325] [mindie-server] [ERROR] [endpoint.cpp:53] : [MIE04E02011C] [endpoint] Failed to init engine! Please check in the mindservice.log, pythonlog.log or console output.
[2025-03-10 20:26:36.605+08:00] [324] [A800T-A2-1] [system] [server] [start mindie server] [fail]
[2025-03-10 20:26:47.072+08:00] [324] [A800T-A2-1] [system] [server] [start endpoint] [fail]
[2025-03-10 20:26:47.073+08:00] [324] [A800T-A2-1] [system] [server] [start mindie server] [fail]
ERR: Failed to init endpoint! Please check the service log or console output.

3) 查询容器的共享内存大小为64M,原因是拉起容器时未指定--shm-size参数,默认只会给容器分配64M共享内存

df -h /dev/shm

【问题根因】

拉起容器时未指定--shm-size参数,默认只给容器分配64M共享内存。容器的共享内存不足导致MindIE服务启动失败

【解决方案】

方案一:创建容器时添加参数选项--shm-size=1G

方案二:(不推荐)创建容器时添加参数选项--ipc=host,当使用 --ipc=host 运行容器时,容器会直接使用宿主机的 /dev/shm,而不是创建一个独立的 /dev/shm 供容器使用。这就意味着容器内的进程可以直接访问和操作宿主机 /dev/shm 下的共享内存对象,反之,宿主机上的进程也能感知到容器内进程对 /dev/shm 所做的操作

【相关知识】

默认情况下,容器的共享内存大小被限制为 64MB。如果需要修改容器共享内存的默认大小,可以编辑 /etc/docker/daemon.json 文件,添加或修改 "default-shm-size" 字段,例如将容器共享内存的默认大小设置为 256MB。修改完成后,需要重启 Docker 服务systemctl restart docker使配置生效

{
    "default-shm-size": "256m"
}

我要发帖子