模型：Qwen2.5-32B-Instruct-AWQ.
好像是以下错误：
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
ConnectionRefusedError: [Errno 111] Connection refused
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:62] : [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
错误详细信息如下
[root@autodl-container-fc57458855-494f1172 mindie-service]# cd /usr/local/Ascend/mindie/latest/mindie-service/bin
[root@autodl-container-fc57458855-494f1172 bin]# ./mindieservice_daemon
The old environment variable ATB_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
2025-07-18 11:51:24.230 10900 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu] [processid] [threadid] [llm] [loglevel] [file:line] [status code] msg
[2025-07-18 11:51:24.230] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:76] LLMRuntime init success!
[2025-07-18 11:51:29.619] [10898] [281473664081888] [llm] [INFO] [model_deploy_config.cpp:281] ModelDeployConfig::vocabSize=152064
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:201] InitModelConfig: maxSeqLen=2560, maxInputTokenLen=2048, g_truncation=0, g_truncLen=1
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:263] Input plugin_params is empty or only contains whitespace.
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:263] Input plugin_params is empty or only contains whitespace.
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:436] The rankIdList is In InitEngineConfig paras
Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.
[2025-07-18 11:51:29.624+08:00] [10898] [10900] [batchscheduler] ===init log===
[2025-07-18 11:51:29.626+08:00] [10898] ===init log===
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:319] : [model_backend] Init Instance with config map:
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:132] : [model_backend] {asyncBatchscheduler, false},{async_infer, false},{backend_bin_path, /usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/},{backend_log_file, /usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/},{backend_modelInstance_id, 0},{backend_type, atb},{block_size, 128},{cpu_mem, 5},{deploy_type, INTER_PROCESS},{distributed_enable, false},{executor_type, LLM_EXECUTOR_PYTHON},{globalRankIds, },{globalWorldSize, 0},{infer_mode, standard},{interNodeKmcKsfMaster, tools/pmt/master/ksfa},{interNodeKmcKsfStandby, tools/pmt/standby/ksfb},{interNodeTLSEnabled, 1},{interNodeTlsCaFiles, ca.pem,},{interNodeTlsCaPath, security/grpc/ca/},{interNodeTlsCert, security/grpc/certs/server.pem},{interNodeTlsCrlFiles, server_crl.pem,},{interNodeTlsCrlPath, security/grpc/certs/},{interNodeTlsPk, security/grpc/keys/server.key.pem},{interNodeTlsPkPwd, security/grpc/pass/mindie_server_key_pwd.txt},{isMaster, 0},{is_dmi_infer, 0},{localIP, },{log_error, 1},{log_file_num, 20},{log_file_size, 20},{log_info, 1},{log_verbose, 0},{log_warning, 1},{masterIP, },{max_batch_size, 200},{max_input_len, 2048},{max_iter_times, 512},{max_prefill_tokens, 8192},{max_seq_len, 2560},{model_id, /root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ},{model_instance_number, 1},{model_instance_type, Standard},{model_name, qwen2.5},{multiNodesInferEnabled, 0},{multiNodesInferPort, 1120},{npu_device_ids, 0},{npu_mem, -1},{slaveIPs, },{speculation_gamma, 0},{threadNum, 1},{trust_remote_code, 0},{world_size, 1},
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [utils.cpp:324] : [model_backend] Close async scheduling.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [connector_launcher.cpp:155] : [model_backend] Signal SIGCHLD handler has been registered.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112640MB.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.649+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112608MB.
[2025-07-18 11:51:29.649+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.670+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112576MB.
[2025-07-18 11:51:29.670+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.692+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112544MB.
[2025-07-18 11:51:29.692+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.728+08:00] [10898] [10900] [batchscheduler] [INFO] [connector_launcher.cpp:212] : [model_backend] Launch one sub process 11117 for process group: 10898
Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.
[2025-07-18 11:51:29.752+08:00] [11117] [11117] [batchscheduler] ===init log===
[2025-07-18 11:51:29.755+08:00] [11117] ===init log===
[2025-07-18 11:51:29.755+08:00] [11117] [11117] [batchscheduler] [INFO] [slave_IPC_communicator.cpp:1174] : [model_backend] Slave communicator launch threads success.
[2025-07-18 11:51:29.755+08:00] [11117] [autodl-container-fc57458855-494f1172] [system] [model_backend] [create slave backend connector subprocess] [success]
[msservice_profiler] [PID:11117] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:11117] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[2025-07-18 11:51:30.728+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:373] : [model_backend] InstanceInit isMultiNodesInfer is 0
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [model.py:84] : [python thread: infer] launch do_inference thread. thread_num: 1
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:16] : model_config {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item local_rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item infer_mode value is standard.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item model_instance_type value is Standard.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item speculation_gamma value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item backend_type value is atb.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named plugin_params, use default value .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item plugin_params value is .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named tp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item tp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named pp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item pp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named ep, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item ep value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named dp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item dp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named es, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item es value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named sp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item sp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item model_id value is /root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item local_rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item globalRankIds value is .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item globalWorldSize value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item world_size value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item npu_device_id value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:84] : The list item npu_device_ids value is ['0'].
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item cpu_mem value is 5.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item npu_mem value is -1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_seq_len value is 2560.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_iter_times value is 512.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_prefill_tokens value is 8192.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item block_size value is 128.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item distributed_enable value is false.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:24] : >>model_config after initialize {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [model.py:41] : init base config
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [metrics.py:56] : profiling is disenabled.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [standard_model.py:132] : global rank id 0 get model config: {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
The old environment variable ATB_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
[2025-07-18 11:51:37,462] [11117] [281469620515168] [llmmodels] [INFO] [cpu_binding.py-254] : rank_id: 0, device_id: 0, numa_id: 6, shard_devices: [0], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167]
[2025-07-18 11:51:37,465] [11117] [281469620515168] [llmmodels] [INFO] [cpu_binding.py-280] : process 11117, new_affinity is [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167], cpu count 24
[2025-07-18 11:51:38,216] [11117] [281469620515168] [llmmodels] [INFO] [model_runner.py-154] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-07-18 11:51:44,103] [11117] [281469620515168] [llmmodels] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-07-18 11:51:44,104] [11117] [281469620515168] [llmmodels] [INFO] [model_runner.py-176] : init tokenizer done
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:62] : [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 60, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 189, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 31, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 133, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 135, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 79, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 203, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 29, in __init__
super().__init__(config, weights, **kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 109, in __init__
self.placeholder = torch.zeros(1, dtype=self.dtype, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:65] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'}
[2025-07-18 11:51:44.429+08:00] [10898] [10900] [server] [WARN] [llm_daemon.cpp:74] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-07-18 11:51:44.429+08:00] [10898] [10900] [server] [ERROR] [llm_daemon.cpp:90] : [MIE04E010109] [daemon] Process 11117 was terminated by signal 9 (Killed)
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-6:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
点赞
收藏
回复
分享
举报
已解决
模型：Qwen2.5-32B-Instruct-AWQ.
好像是以下错误：
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
ConnectionRefusedError: [Errno 111] Connection refused
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:62] : [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
错误详细信息如下
[root@autodl-container-fc57458855-494f1172 mindie-service]# cd /usr/local/Ascend/mindie/latest/mindie-service/bin
[root@autodl-container-fc57458855-494f1172 bin]# ./mindieservice_daemon
The old environment variable ATB_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
2025-07-18 11:51:24.230 10900 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu] [processid] [threadid] [llm] [loglevel] [file:line] [status code] msg
[2025-07-18 11:51:24.230] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:76] LLMRuntime init success!
[2025-07-18 11:51:29.619] [10898] [281473664081888] [llm] [INFO] [model_deploy_config.cpp:281] ModelDeployConfig::vocabSize=152064
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:201] InitModelConfig: maxSeqLen=2560, maxInputTokenLen=2048, g_truncation=0, g_truncLen=1
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:263] Input plugin_params is empty or only contains whitespace.
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:263] Input plugin_params is empty or only contains whitespace.
[2025-07-18 11:51:29.623] [10898] [281473664081888] [llm] [INFO] [llm_manager_impl.cpp:436] The rankIdList is In InitEngineConfig paras
Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.
[2025-07-18 11:51:29.624+08:00] [10898] [10900] [batchscheduler] ===init log===
[2025-07-18 11:51:29.626+08:00] [10898] ===init log===
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:319] : [model_backend] Init Instance with config map:
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:132] : [model_backend] {asyncBatchscheduler, false},{async_infer, false},{backend_bin_path, /usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/},{backend_log_file, /usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/},{backend_modelInstance_id, 0},{backend_type, atb},{block_size, 128},{cpu_mem, 5},{deploy_type, INTER_PROCESS},{distributed_enable, false},{executor_type, LLM_EXECUTOR_PYTHON},{globalRankIds, },{globalWorldSize, 0},{infer_mode, standard},{interNodeKmcKsfMaster, tools/pmt/master/ksfa},{interNodeKmcKsfStandby, tools/pmt/standby/ksfb},{interNodeTLSEnabled, 1},{interNodeTlsCaFiles, ca.pem,},{interNodeTlsCaPath, security/grpc/ca/},{interNodeTlsCert, security/grpc/certs/server.pem},{interNodeTlsCrlFiles, server_crl.pem,},{interNodeTlsCrlPath, security/grpc/certs/},{interNodeTlsPk, security/grpc/keys/server.key.pem},{interNodeTlsPkPwd, security/grpc/pass/mindie_server_key_pwd.txt},{isMaster, 0},{is_dmi_infer, 0},{localIP, },{log_error, 1},{log_file_num, 20},{log_file_size, 20},{log_info, 1},{log_verbose, 0},{log_warning, 1},{masterIP, },{max_batch_size, 200},{max_input_len, 2048},{max_iter_times, 512},{max_prefill_tokens, 8192},{max_seq_len, 2560},{model_id, /root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ},{model_instance_number, 1},{model_instance_type, Standard},{model_name, qwen2.5},{multiNodesInferEnabled, 0},{multiNodesInferPort, 1120},{npu_device_ids, 0},{npu_mem, -1},{slaveIPs, },{speculation_gamma, 0},{threadNum, 1},{trust_remote_code, 0},{world_size, 1},
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [utils.cpp:324] : [model_backend] Close async scheduling.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [connector_launcher.cpp:155] : [model_backend] Signal SIGCHLD handler has been registered.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112640MB.
[2025-07-18 11:51:29.626+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.649+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112608MB.
[2025-07-18 11:51:29.649+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.670+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112576MB.
[2025-07-18 11:51:29.670+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.692+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:181] : [model_backend] Total shared memory size:112640MB, and available shared memory size:112544MB.
[2025-07-18 11:51:29.692+08:00] [10898] [10900] [batchscheduler] [INFO] [share_memory.cpp:182] : [model_backend] Shared memory size check success.
[2025-07-18 11:51:29.728+08:00] [10898] [10900] [batchscheduler] [INFO] [connector_launcher.cpp:212] : [model_backend] Launch one sub process 11117 for process group: 10898
Log default log dir is ~/mindie/log, your can use env MINDIE_LOG_PATH to change log saving dir.
[2025-07-18 11:51:29.752+08:00] [11117] [11117] [batchscheduler] ===init log===
[2025-07-18 11:51:29.755+08:00] [11117] ===init log===
[2025-07-18 11:51:29.755+08:00] [11117] [11117] [batchscheduler] [INFO] [slave_IPC_communicator.cpp:1174] : [model_backend] Slave communicator launch threads success.
[2025-07-18 11:51:29.755+08:00] [11117] [autodl-container-fc57458855-494f1172] [system] [model_backend] [create slave backend connector subprocess] [success]
[msservice_profiler] [PID:11117] [DEBUG] [ReadEnable:344] profile enable_: false
[msservice_profiler] [PID:11117] [DEBUG] [ReadAclTaskTime:372] profile enableAclTaskTime_: false
[2025-07-18 11:51:30.728+08:00] [10898] [10900] [batchscheduler] [INFO] [model_backend.cpp:373] : [model_backend] InstanceInit isMultiNodesInfer is 0
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [model.py:84] : [python thread: infer] launch do_inference thread. thread_num: 1
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:16] : model_config {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item local_rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item infer_mode value is standard.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item model_instance_type value is Standard.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item speculation_gamma value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item backend_type value is atb.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named plugin_params, use default value .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item plugin_params value is .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named tp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item tp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named pp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item pp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named ep, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item ep value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named dp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item dp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named es, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item es value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:68] : There is no item named sp, use default value 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item sp value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item model_id value is /root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item local_rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item rank value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item globalRankIds value is .
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item globalWorldSize value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item world_size value is 1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item npu_device_id value is 0.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:84] : The list item npu_device_ids value is ['0'].
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item cpu_mem value is 5.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item npu_mem value is -1.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_seq_len value is 2560.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_iter_times value is 512.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item max_prefill_tokens value is 8192.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item block_size value is 128.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:72] : The item distributed_enable value is false.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [config.py:24] : >>model_config after initialize {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [model.py:41] : init base config
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [metrics.py:56] : profiling is disenabled.
[2025-07-18 11:51:35.018+0800] [11117] [281472845869408] [batchscheduler] [INFO] [standard_model.py:132] : global rank id 0 get model config: {'asyncBatchscheduler': 'false', 'async_infer': 'false', 'backend_bin_path': '/usr/local/Ascend/mindie/2.0.RC2/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/2.0.RC2/mindie-service/~/mindie/log/', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'distributed_enable': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'infer_mode': 'standard', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'is_dmi_infer': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_batch_size': '200', 'max_input_len': '2048', 'max_iter_times': '512', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_id': '/root/autodl-tmp/Qwen/Qwen2.5-32B-Instruct-AWQ', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2.5', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'threadNum': '1', 'trust_remote_code': '0', 'world_size': '1'}
The old environment variable ATB_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable ATB_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_PATH will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_PATH as soon as possible.
The old environment variable ATB_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable MINDIE_LLM_PYTHON_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable MINDIE_LLM_LOG_TO_FILE will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_FILE as soon as possible.
The old environment variable OCK_LOG_LEVEL will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_LEVEL as soon as possible.
The old environment variable OCK_LOG_TO_STDOUT will be deprecated on 2025/12/31. Please use the new environment variable MINDIE_LOG_TO_STDOUT as soon as possible.
[2025-07-18 11:51:37,462] [11117] [281469620515168] [llmmodels] [INFO] [cpu_binding.py-254] : rank_id: 0, device_id: 0, numa_id: 6, shard_devices: [0], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167]
[2025-07-18 11:51:37,465] [11117] [281469620515168] [llmmodels] [INFO] [cpu_binding.py-280] : process 11117, new_affinity is [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167], cpu count 24
[2025-07-18 11:51:38,216] [11117] [281469620515168] [llmmodels] [INFO] [model_runner.py-154] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-07-18 11:51:44,103] [11117] [281469620515168] [llmmodels] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-07-18 11:51:44,104] [11117] [281469620515168] [llmmodels] [INFO] [model_runner.py-176] : init tokenizer done
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:62] : [Model] >>> Exception:call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 60, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 189, in __init__
self.generator_backend = get_generator_backend(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/__init__.py", line 31, in get_generator_backend
return generator_cls(model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 133, in __init__
super().__init__(model_config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 135, in __init__
self.model_wrapper = get_model_wrapper(model_config, backend_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/__init__.py", line 15, in get_model_wrapper
return wrapper_cls(**model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 79, in __init__
self.model_runner.load_weights()
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 203, in load_weights
self.model = self.model_cls(self.config,
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 29, in __init__
super().__init__(config, weights, **kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 109, in __init__
self.placeholder = torch.zeros(1, dtype=self.dtype, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 11117] 2025-07-18-11:51:44.230.506 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-07-18-11:51:44 (PID:11117, Device:0, RankID:-1) ERR01100 OPS call acl api failed
[2025-07-18 11:51:44.018+0800] [11117] [281472845869408] [batchscheduler] [ERROR] [model.py:65] : [MIE04E13030A] [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'}
[2025-07-18 11:51:44.429+08:00] [10898] [10900] [server] [WARN] [llm_daemon.cpp:74] : [MIE04W01011A] [daemon] Received exit signal[17]
[2025-07-18 11:51:44.429+08:00] [10898] [10900] [server] [ERROR] [llm_daemon.cpp:90] : [MIE04E010109] [daemon] Process 11117 was terminated by signal 9 (Killed)
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Process ForkServerProcess-6:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 814, in _callmethod
conn = self._tls.connection
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 71, in wrapper
raise exp
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 63, in wrapper
func(*args, **kwargs)
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 264, in task_distribute
resource_proxy[SUB_PROCESS_STATE].append(True)
File "<string>", line 2, in append
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 818, in _callmethod
self._connect()
File "/usr/lib64/python3.11/multiprocessing/managers.py", line 805, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 518, in Client
c = SocketClient(address)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/multiprocessing/connection.py", line 646, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed