A800IA2(64G)
镜像版本： swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts
推理原版Qwen2.5没问题，推理SFT后的有问题
tokenizer校验了没问题，transformer没问题
环境变量已经配置
export HCCL_DETERMINISTIC=false
export HCCL_OP_EXPANSION_MODE="AIV"
以下是收集的日志
[root@AI-GPU-SERVER1 mindie-service]# ./bin/mindieservice_daemon
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
Killed
# MindIE Service日志
/usr/local/Ascend/mindie/latest/mindie-service/logs/cat mindie_audit.log
[2025-02-18 10:24:51.066276][111][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:24:51.066377][111][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:25:01.793357][111][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:25:01.793389][111][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:28:44.222839][314][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:28:44.222933][314][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:28:54.983731][314][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:28:54.983757][314][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:32:26.811310][517][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:32:26.811419][517][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:32:37.301190][517][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:32:37.301220][517][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:43:09.985604][109][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:43:09.985656][109][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:43:20.473939][109][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:43:20.473967][109][AI-GPU-SERVER1][system][start][mindie server][fail]
/usr/local/Ascend/mindie/latest/mindie-service/logs/mindservice.log
[root@AI-GPU-SERVER1 logs]# cat mindservice.log
[2025-02-18 10:24:51.066071+08:00] [111] [113] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:24:51.066186+08:00] [111] [113] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:24:51.066248+08:00] [111] [113] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:24:51.066339+08:00] [111] [113] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:24:51.066367+08:00] [111] [113] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:28:44.222609+08:00] [314] [316] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:28:44.222748+08:00] [314] [316] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:28:44.222811+08:00] [314] [316] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:28:44.222892+08:00] [314] [316] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:28:44.222923+08:00] [314] [316] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:32:26.811175+08:00] [517] [519] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:32:26.811227+08:00] [517] [519] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:32:26.811283+08:00] [517] [519] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:32:26.811380+08:00] [517] [519] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:32:26.811409+08:00] [517] [519] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:43:09.985459+08:00] [109] [111] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:43:09.985517+08:00] [109] [111] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:43:09.985575+08:00] [109] [111] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:43:09.985633+08:00] [109] [111] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:43:09.985647+08:00] [109] [111] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
#Ascend
/root/ascend/log/run/plog/cat plog-109_20250218104308865.log
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:08.865.077 [prof_atls_plugin.cpp:117] (tid:111) RegisterProfileCallback, callback type is 7
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:08.865.257 [prof_atls_plugin.cpp:117] (tid:111) RegisterProfileCallback, callback type is 8
[INFO] ATRACE(109,mindieservice_daemon):2025-02-18-10:43:08.866.139 [trace_attr.c:188](tid:111) attr init success, timeout=0ms.
[INFO] ATRACE(109,mindieservice_daemon):2025-02-18-10:43:08.866.164 [trace_recorder.c:121](tid:111) use root path: /root/ascend
[INFO] RUNTIME(109,mindieservice_daemon):2025-02-18-10:43:08.870.137 [task_fail_callback_manager.cc:52] 111 TaskFailCallBackManager: Constructor.
[INFO] HCCL(109,mindieservice_daemon):2025-02-18-10:43:08.946.838 [adapter_rts.cc:2646][111][adapter_rts.cc][CallBackInitRts] g_deviceType [6] g_deviceLogicId [-1] g_devicePhyId [-1]
[INFO] RUNTIME(109,mindieservice_daemon):2025-02-18-10:43:09.063.578 [runtime.cc:5471] 111 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:09.066.986 [prof_atls_plugin.cpp:210] (tid:111) Module[7] register callback of ctrl handle.
点赞
收藏
回复
分享
举报
已解决
A800IA2(64G)
镜像版本： swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-800I-A2-py311-openeuler24.03-lts
推理原版Qwen2.5没问题，推理SFT后的有问题
tokenizer校验了没问题，transformer没问题
环境变量已经配置
export HCCL_DETERMINISTIC=false
export HCCL_OP_EXPANSION_MODE="AIV"
以下是收集的日志
[root@AI-GPU-SERVER1 mindie-service]# ./bin/mindieservice_daemon
LLMInferEngine failed to init LLMInferModels
ERR: Failed to init endpoint! Please check the service log or console output.
Killed
# MindIE Service日志
/usr/local/Ascend/mindie/latest/mindie-service/logs/cat mindie_audit.log
[2025-02-18 10:24:51.066276][111][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:24:51.066377][111][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:25:01.793357][111][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:25:01.793389][111][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:28:44.222839][314][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:28:44.222933][314][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:28:54.983731][314][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:28:54.983757][314][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:32:26.811310][517][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:32:26.811419][517][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:32:37.301190][517][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:32:37.301220][517][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:43:09.985604][109][AI-GPU-SERVER1][system][start][engine(mindieservice_llm_engine)][fail]
[2025-02-18 10:43:09.985656][109][AI-GPU-SERVER1][system][start][mindie server][fail]
[2025-02-18 10:43:20.473939][109][AI-GPU-SERVER1][system][start][endpoint][fail]
[2025-02-18 10:43:20.473967][109][AI-GPU-SERVER1][system][start][mindie server][fail]
/usr/local/Ascend/mindie/latest/mindie-service/logs/mindservice.log
[root@AI-GPU-SERVER1 logs]# cat mindservice.log
[2025-02-18 10:24:51.066071+08:00] [111] [113] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:24:51.066186+08:00] [111] [113] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:24:51.066248+08:00] [111] [113] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:24:51.066339+08:00] [111] [113] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:24:51.066367+08:00] [111] [113] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:28:44.222609+08:00] [314] [316] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:28:44.222748+08:00] [314] [316] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:28:44.222811+08:00] [314] [316] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:28:44.222892+08:00] [314] [316] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:28:44.222923+08:00] [314] [316] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:32:26.811175+08:00] [517] [519] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:32:26.811227+08:00] [517] [519] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:32:26.811283+08:00] [517] [519] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:32:26.811380+08:00] [517] [519] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:32:26.811409+08:00] [517] [519] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
[2025-02-18 10:43:09.985459+08:00] [109] [111] [server] [ERROR] [llm_infer_model_instance.cpp:234] : [LLMInferModelInstance::Init] llmManager_ init fail!
[2025-02-18 10:43:09.985517+08:00] [109] [111] [server] [ERROR] [llm_infer_model.cpp:28] : ModelInstance init failed
[2025-02-18 10:43:09.985575+08:00] [109] [111] [server] [ERROR] [llm_infer_engine.cpp:48] : [MIE05E040001] [llm_backend] LLMInferEngine initialize LLMInferModels failed. please check the error stacks for more details
[2025-02-18 10:43:09.985633+08:00] [109] [111] [server] [ERROR] [infer_backend_manager.cpp:272] : [MIE04E060108] [backend_manager] Failed to init engine: mindieservice_llm_engine
[2025-02-18 10:43:09.985647+08:00] [109] [111] [server] [ERROR] [EndPoint endpoint.cpp:50] ERR: Failed to init backendManager! Please check in the mindservice.log, pythonlog.log(default path: mindie-llm/logs) or console output.
#Ascend
/root/ascend/log/run/plog/cat plog-109_20250218104308865.log
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:08.865.077 [prof_atls_plugin.cpp:117] (tid:111) RegisterProfileCallback, callback type is 7
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:08.865.257 [prof_atls_plugin.cpp:117] (tid:111) RegisterProfileCallback, callback type is 8
[INFO] ATRACE(109,mindieservice_daemon):2025-02-18-10:43:08.866.139 [trace_attr.c:188](tid:111) attr init success, timeout=0ms.
[INFO] ATRACE(109,mindieservice_daemon):2025-02-18-10:43:08.866.164 [trace_recorder.c:121](tid:111) use root path: /root/ascend
[INFO] RUNTIME(109,mindieservice_daemon):2025-02-18-10:43:08.870.137 [task_fail_callback_manager.cc:52] 111 TaskFailCallBackManager: Constructor.
[INFO] HCCL(109,mindieservice_daemon):2025-02-18-10:43:08.946.838 [adapter_rts.cc:2646][111][adapter_rts.cc][CallBackInitRts] g_deviceType [6] g_deviceLogicId [-1] g_devicePhyId [-1]
[INFO] RUNTIME(109,mindieservice_daemon):2025-02-18-10:43:09.063.578 [runtime.cc:5471] 111 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set
[INFO] PROFILING(109,mindieservice_daemon):2025-02-18-10:43:09.066.986 [prof_atls_plugin.cpp:210] (tid:111) Module[7] register callback of ctrl handle.