按照官方部署指引，部署deepseek 32B版本，
DeepSeek-R1-Distill-Qwen-32B-模型库-ModelZoo-昇腾社区
但是报下述错误：
说明：（同样的配置文件，将模型路径改成14B的模型后， 就成功了，不知道为啥， 插了4块 300I pro的NPU卡，理论上显存是够的）
[2025-02-17 10:35:56,402] [119604] [281466135626080] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,754] [119600] [281465480016224] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,757] [119600] [281465480016224] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,757] [119600] [281465480016224] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473061412432 id of vcache is 281473049672880
[2025-02-17 10:35:57,185] [119611] [281465273500000] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:57.840480] [error] [120278] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:0, rankSize:4
[2025-02-17 10:35:57.840547] [error] [120278] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:57.840558] [error] [120278] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:0
[2025-02-17 10:35:57.846170] [error] [121106] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 0
[2025-02-17 10:35:57.846193] [error] [121106] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:57.846204] [error] [121106] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:57.846214] [error] [121106] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:57.846222] [error] [121106] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:57.846230] [error] [121106] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
2025-02-17 10:35:57,847 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
2025-02-17 10:35:57,850 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-02-17 10:35:58.20077] [error] [120277] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:3, rankSize:4
[2025-02-17 10:35:58.20069] [error] [120279] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:2, rankSize:4
[2025-02-17 10:35:58.20139] [error] [120277] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.20150] [error] [120279] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.20150] [error] [120277] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:3
[2025-02-17 10:35:58.20162] [error] [120279] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:2
[2025-02-17 10:35:58.22052] [error] [120276] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:1, rankSize:4
[2025-02-17 10:35:58.22126] [error] [120276] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.22138] [error] [120276] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:1
[2025-02-17 10:35:58.26184] [error] [121145] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 2
[2025-02-17 10:35:58.26218] [error] [121145] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26232] [error] [121145] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.26243] [error] [121145] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26253] [error] [121145] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.26261] [error] [121145] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
[2025-02-17 10:35:58.26409] [error] [121108] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 3
[2025-02-17 10:35:58.26446] [error] [121108] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26459] [error] [121108] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.26470] [error] [121108] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26482] [error] [121108] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.26489] [error] [121108] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
[2025-02-17 10:35:58.28069] [error] [121104] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 1
[2025-02-17 10:35:58.28095] [error] [121104] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.28108] [error] [121104] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.28118] [error] [121104] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.28127] [error] [121104] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.28131] [error] [121104] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
2025-02-17 10:35:58,027 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
2025-02-17 10:35:58,031 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-02-17 10:35:58,027 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
配置文件如下（跑14B没问题，改成32B的模型路径就出现上述问题）：
cat conf/config.json
{
"Version" : "1.1.0",
"LogConfig" :
{
"logLevel" : "Info",
"logFileSize" : 20,
"logFileNum" : 20,
"logPath" : "logs/mindservice.log"
},
"ServerConfig" :
{
"ipAddress" : "10.201.201.111",
"managementIpAddress" : "127.0.0.2",
"port" : 1050,
"managementPort" : 1051,
"metricsPort" : 1052,
"allowAllZeroIpListening" : false,
"maxLinkNum" : 1000,
"httpsEnabled" : false,
"fullTextEnabled" : false,
"tlsCaPath" : "security/ca/",
"tlsCaFile" : ["ca.pem"],
"tlsCert" : "security/certs/server.pem",
"tlsPk" : "security/keys/server.key.pem",
"tlsPkPwd" : "security/pass/key_pwd.txt",
"tlsCrlPath" : "security/certs/",
"tlsCrlFiles" : ["server_crl.pem"],
"managementTlsCaFile" : ["management_ca.pem"],
"managementTlsCert" : "security/certs/management/server.pem",
"managementTlsPk" : "security/keys/management/server.key.pem",
"managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
"managementTlsCrlPath" : "security/management/certs/",
"managementTlsCrlFiles" : ["server_crl.pem"],
"kmcKsfMaster" : "tools/pmt/master/ksfa",
"kmcKsfStandby" : "tools/pmt/standby/ksfb",
"inferMode" : "standard",
"interCommTLSEnabled" : true,
"interCommPort" : 1121,
"interCommTlsCaPath" : "security/grpc/ca/",
"interCommTlsCaFiles" : ["ca.pem"],
"interCommTlsCert" : "security/grpc/certs/server.pem",
"interCommPk" : "security/grpc/keys/server.key.pem",
"interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
"interCommTlsCrlPath" : "security/grpc/certs/",
"interCommTlsCrlFiles" : ["server_crl.pem"],
"openAiSupport" : "vllm"
},
"BackendConfig" : {
"backendName" : "mindieservice_llm_engine",
"modelInstanceNumber" : 1,
"npuDeviceIds" : [[0,1,2,3]],
"tokenizerProcessNumber" : 8,
"multiNodesInferEnabled" : false,
"multiNodesInferPort" : 1120,
"interNodeTLSEnabled" : true,
"interNodeTlsCaPath" : "security/grpc/ca/",
"interNodeTlsCaFiles" : ["ca.pem"],
"interNodeTlsCert" : "security/grpc/certs/server.pem",
"interNodeTlsPk" : "security/grpc/keys/server.key.pem",
"interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
"interNodeTlsCrlPath" : "security/grpc/certs/",
"interNodeTlsCrlFiles" : ["server_crl.pem"],
"interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
"interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
"ModelDeployConfig" :
{
"maxSeqLen" : 8192,
"maxInputTokenLen" : 5000,
"truncation" : false,
"ModelConfig" : [
{
"modelInstanceType" : "Standard",
"modelName" : "qwen",
"modelWeightPath" : "/data/deepseek/deepseek-32B",
"worldSize" : 4,
"cpuMemSize" : 5,
"npuMemSize" : -1,
"backendType" : "atb",
"trustRemoteCode" : false
}
]
},
"ScheduleConfig" :
{
"templateType" : "Standard",
"templateName" : "Standard_LLM",
"cacheBlockSize" : 128,
"maxPrefillBatchSize" : 50,
"maxPrefillTokens" : 8192,
"prefillTimeMsPerReq" : 150,
"prefillPolicyType" : 0,
"decodeTimeMsPerReq" : 50,
"decodePolicyType" : 0,
"maxBatchSize" : 200,
"maxIterTimes" : 8192,
"maxPreemptCount" : 0,
"supportSelectBatch" : false,
"maxQueueDelayMicroseconds" : 5000
}
}
}
点赞
收藏
回复
分享
举报
已解决
按照官方部署指引，部署deepseek 32B版本，
DeepSeek-R1-Distill-Qwen-32B-模型库-ModelZoo-昇腾社区
但是报下述错误：
说明：（同样的配置文件，将模型路径改成14B的模型后， 就成功了，不知道为啥， 插了4块 300I pro的NPU卡，理论上显存是够的）
[2025-02-17 10:35:56,402] [119604] [281466135626080] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,754] [119600] [281465480016224] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,757] [119600] [281465480016224] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:56,757] [119600] [281465480016224] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473061412432 id of vcache is 281473049672880
[2025-02-17 10:35:57,185] [119611] [281465273500000] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-02-17 10:35:57.840480] [error] [120278] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:0, rankSize:4
[2025-02-17 10:35:57.840547] [error] [120278] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:57.840558] [error] [120278] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:0
[2025-02-17 10:35:57.846170] [error] [121106] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 0
[2025-02-17 10:35:57.846193] [error] [121106] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:57.846204] [error] [121106] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:57.846214] [error] [121106] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:57.846222] [error] [121106] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:57.846230] [error] [121106] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
2025-02-17 10:35:57,847 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
2025-02-17 10:35:57,850 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[2025-02-17 10:35:58.20077] [error] [120277] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:3, rankSize:4
[2025-02-17 10:35:58.20069] [error] [120279] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:2, rankSize:4
[2025-02-17 10:35:58.20139] [error] [120277] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.20150] [error] [120279] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.20150] [error] [120277] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:3
[2025-02-17 10:35:58.20162] [error] [120279] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:2
[2025-02-17 10:35:58.22052] [error] [120276] [hccl_runner.cpp:132] AllGatherHcclRunner:0 HcclCommInitRootInfo fail, error:2, rank:1, rankSize:4
[2025-02-17 10:35:58.22126] [error] [120276] [comm_pool.h:43] CommPool commCreateFunc fail
[2025-02-17 10:35:58.22138] [error] [120276] [hccl_runner.cpp:72] AllGatherHcclRunner:0 get hccl comm fail by rank:1
[2025-02-17 10:35:58.26184] [error] [121145] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 2
[2025-02-17 10:35:58.26218] [error] [121145] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26232] [error] [121145] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.26243] [error] [121145] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26253] [error] [121145] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.26261] [error] [121145] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
[2025-02-17 10:35:58.26409] [error] [121108] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 3
[2025-02-17 10:35:58.26446] [error] [121108] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26459] [error] [121108] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.26470] [error] [121108] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.26482] [error] [121108] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.26489] [error] [121108] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
[2025-02-17 10:35:58.28069] [error] [121104] [all_gather_hccl_runner.cpp:38] hcclComm is null, rank: 1
[2025-02-17 10:35:58.28095] [error] [121104] [runner.cpp:115] AllGatherHcclRunner_1_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.28108] [error] [121104] [graph_runner.cpp:906] WordEmbeddingRunner_1:0 node[1] execute fail, runner name:AllGatherHcclRunner
[2025-02-17 10:35:58.28118] [error] [121104] [runner.cpp:115] WordEmbeddingRunner_1:1 Execute Failed. st: 3
[2025-02-17 10:35:58.28127] [error] [121104] [operation_base.cpp:726] WordEmbedding_1 execute WordEmbeddingRunner fail
[2025-02-17 10:35:58.28131] [error] [121104] [operation_base.cpp:764] WordEmbedding_1 execute fail, error code: 3
2025-02-17 10:35:58,027 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 303, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 296, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode, dummy=True)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward
logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator
acl_model_out = model_operation.execute(acl_inputs, acl_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
2025-02-17 10:35:58,031 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
2025-02-17 10:35:58,027 [ERROR] model.py:39 - [Model] >>> Exception:Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.
配置文件如下（跑14B没问题，改成32B的模型路径就出现上述问题）：
cat conf/config.json
{
"Version" : "1.1.0",
"LogConfig" :
{
"logLevel" : "Info",
"logFileSize" : 20,
"logFileNum" : 20,
"logPath" : "logs/mindservice.log"
},
"ServerConfig" :
{
"ipAddress" : "10.201.201.111",
"managementIpAddress" : "127.0.0.2",
"port" : 1050,
"managementPort" : 1051,
"metricsPort" : 1052,
"allowAllZeroIpListening" : false,
"maxLinkNum" : 1000,
"httpsEnabled" : false,
"fullTextEnabled" : false,
"tlsCaPath" : "security/ca/",
"tlsCaFile" : ["ca.pem"],
"tlsCert" : "security/certs/server.pem",
"tlsPk" : "security/keys/server.key.pem",
"tlsPkPwd" : "security/pass/key_pwd.txt",
"tlsCrlPath" : "security/certs/",
"tlsCrlFiles" : ["server_crl.pem"],
"managementTlsCaFile" : ["management_ca.pem"],
"managementTlsCert" : "security/certs/management/server.pem",
"managementTlsPk" : "security/keys/management/server.key.pem",
"managementTlsPkPwd" : "security/pass/management/key_pwd.txt",
"managementTlsCrlPath" : "security/management/certs/",
"managementTlsCrlFiles" : ["server_crl.pem"],
"kmcKsfMaster" : "tools/pmt/master/ksfa",
"kmcKsfStandby" : "tools/pmt/standby/ksfb",
"inferMode" : "standard",
"interCommTLSEnabled" : true,
"interCommPort" : 1121,
"interCommTlsCaPath" : "security/grpc/ca/",
"interCommTlsCaFiles" : ["ca.pem"],
"interCommTlsCert" : "security/grpc/certs/server.pem",
"interCommPk" : "security/grpc/keys/server.key.pem",
"interCommPkPwd" : "security/grpc/pass/key_pwd.txt",
"interCommTlsCrlPath" : "security/grpc/certs/",
"interCommTlsCrlFiles" : ["server_crl.pem"],
"openAiSupport" : "vllm"
},
"BackendConfig" : {
"backendName" : "mindieservice_llm_engine",
"modelInstanceNumber" : 1,
"npuDeviceIds" : [[0,1,2,3]],
"tokenizerProcessNumber" : 8,
"multiNodesInferEnabled" : false,
"multiNodesInferPort" : 1120,
"interNodeTLSEnabled" : true,
"interNodeTlsCaPath" : "security/grpc/ca/",
"interNodeTlsCaFiles" : ["ca.pem"],
"interNodeTlsCert" : "security/grpc/certs/server.pem",
"interNodeTlsPk" : "security/grpc/keys/server.key.pem",
"interNodeTlsPkPwd" : "security/grpc/pass/mindie_server_key_pwd.txt",
"interNodeTlsCrlPath" : "security/grpc/certs/",
"interNodeTlsCrlFiles" : ["server_crl.pem"],
"interNodeKmcKsfMaster" : "tools/pmt/master/ksfa",
"interNodeKmcKsfStandby" : "tools/pmt/standby/ksfb",
"ModelDeployConfig" :
{
"maxSeqLen" : 8192,
"maxInputTokenLen" : 5000,
"truncation" : false,
"ModelConfig" : [
{
"modelInstanceType" : "Standard",
"modelName" : "qwen",
"modelWeightPath" : "/data/deepseek/deepseek-32B",
"worldSize" : 4,
"cpuMemSize" : 5,
"npuMemSize" : -1,
"backendType" : "atb",
"trustRemoteCode" : false
}
]
},
"ScheduleConfig" :
{
"templateType" : "Standard",
"templateName" : "Standard_LLM",
"cacheBlockSize" : 128,
"maxPrefillBatchSize" : 50,
"maxPrefillTokens" : 8192,
"prefillTimeMsPerReq" : 150,
"prefillPolicyType" : 0,
"decodeTimeMsPerReq" : 50,
"decodePolicyType" : 0,
"maxBatchSize" : 200,
"maxIterTimes" : 8192,
"maxPreemptCount" : 0,
"supportSelectBatch" : false,
"maxQueueDelayMicroseconds" : 5000
}
}
}