使用一张atlas300i Duo，按照
https://www.hiascend.com/software/modelzoo/models/detail/2a0d0cb1bc644eee8318a12429de67b6
指引完成了DeepSeek-R1-Distill-Qwen-14B模型的稀疏量化，通过torchrun调用模型正常。然后通过mindieservice_daemon执行推理报错。报错如下：
[root@localhost bin]# vim /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
[root@localhost bin]# ./mindieservice_daemon
2025-03-21 11:15:30,440 [INFO] config.py:30 - model_config {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '2048', 'max_iter_times': '5120', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_cut_policy': 'standard', 'model_id': '/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '0', 'world_size': '1'}
2025-03-21 11:15:30,440 [INFO] config.py:35 - do not init dmi config
2025-03-21 11:15:30,440 [INFO] standard_model.py:145 - rank id 0 get model config: {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '2048', 'max_iter_times': '5120', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_cut_policy': 'standard', 'model_id': '/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '0', 'world_size': '1'}
[2025-03-21 11:15:34,893] [13781] [281459877212512] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-03-21 11:15:34,895] [13781] [281459877212512] [llm] [INFO] [cpu_binding.py-238] : process 13781, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], cpu count 16
[2025-03-21 11:15:35,469] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : model_runner.quantize: w8a8sc, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-03-21 11:15:40,902] [13781] [281459877212512] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-03-21 11:15:40,904] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<\uff5cbegin\u2581of\u2581sentence\uff5c>', 'eos_token': '<\uff5cend\u2581of\u2581sentence\uff5c>', 'pad_token': '<\uff5cend\u2581of\u2581sentence\uff5c>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<\uff5cend\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<\uff5cUser\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<\uff5cAssistant\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<\uff5cbegin\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib64/python3.11/logging/__init__.py", line 1113, in emit
stream.write(msg + self.terminator)
UnicodeEncodeError: 'ascii' codec can't encode character '\uff5c' in position 325: ordinal not in range(128)
Call stack:
[2025-03-21 11:15:40,920] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
File "/usr/lib64/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/utils/log/multiprocess_logging_handler.py", line 80, in receive
self.log_handler.emit(record)
Message: 'init tokenizer done: LlamaTokenizerFast(name_or_path=\'/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc\', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side=\'left\', truncation_side=\'right\', special_tokens={\'bos_token\': \'<\uff5cbegin\u2581of\u2581sentence\uff5c>\', \'eos_token\': \'<\uff5cend\u2581of\u2581sentence\uff5c>\', \'pad_token\': \'<\uff5cend\u2581of\u2581sentence\uff5c>\'}, clean_up_tokenization_spaces=False), added_tokens_decoder={\n\t151643: AddedToken("<\uff5cend\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151644: AddedToken("<\uff5cUser\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151645: AddedToken("<\uff5cAssistant\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151646: AddedToken("<\uff5cbegin\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n}'
Arguments: ()
[2025-03-21 11:15:40,996] [13781] [281459877212512] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-03-21 11:15:45,513] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8SparseCompressedLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8SparseCompressedLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8SparseCompressedLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorHead(
(linear): FastLinear()
)
)
[2025-03-21 11:15:45,571] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,571] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,572] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,572] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,573] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,573] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,574] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,574] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,575] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,576] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,576] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,577] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,579] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,581] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,582] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,582] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,583] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,583] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,584] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,585] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,585] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,586] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,586] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,588] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,590] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,592] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,593] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,593] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,594] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,596] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,598] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,598] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,599] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,599] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,600] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,602] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,603] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,603] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,604] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,604] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,605] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,605] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,606] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,606] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,607] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,607] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,608] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,608] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,609] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,609] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,610] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,610] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,611] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,611] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,612] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,612] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,613] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,613] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,614] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,614] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,615] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,617] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,617] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,618] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,618] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,619] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,619] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,620] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,620] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,621] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,622] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,622] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,623] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,625] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,627] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,628] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,628] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,629] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,629] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,632] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:46,184] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-03-21 11:15:46,197] [13781] [281459877212512] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-03-21 11:15:46,199] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281466043764272 id of vcache is 281466043764176
2025-03-21 11:15:48,541 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-21-11:15:48 (PID:13781, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 13781] 2025-03-21-11:15:48.341.339 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=994, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[13781]:
EXCEPTION STREAM:
Exception info:TGID=1727582, model id=65535, stream id=2, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=14, stream_id=2
Other info[0]:time=2025-03-21-11:15:48.257.677, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-21-11:15:48 (PID:13781, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 13781] 2025-03-21-11:15:48.341.339 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=994, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[13781]:
EXCEPTION STREAM:
Exception info:TGID=1727582, model id=65535, stream id=2, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=14, stream_id=2
Other info[0]:time=2025-03-21-11:15:48.257.677, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-21 11:15:48,543 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
[root@localhost bin]# ^C
[root@localhost bin]#
点赞
收藏
回复
分享
举报
已解决
使用一张atlas300i Duo，按照
https://www.hiascend.com/software/modelzoo/models/detail/2a0d0cb1bc644eee8318a12429de67b6
指引完成了DeepSeek-R1-Distill-Qwen-14B模型的稀疏量化，通过torchrun调用模型正常。然后通过mindieservice_daemon执行推理报错。报错如下：
[root@localhost bin]# vim /usr/local/Ascend/mindie/latest/mindie-service/conf/config.json
[root@localhost bin]# ./mindieservice_daemon
2025-03-21 11:15:30,440 [INFO] config.py:30 - model_config {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '2048', 'max_iter_times': '5120', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_cut_policy': 'standard', 'model_id': '/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '0', 'world_size': '1'}
2025-03-21 11:15:30,440 [INFO] config.py:35 - do not init dmi config
2025-03-21 11:15:30,440 [INFO] standard_model.py:145 - rank id 0 get model config: {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '5', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '2048', 'max_iter_times': '5120', 'max_prefill_tokens': '8192', 'max_seq_len': '2560', 'model_cut_policy': 'standard', 'model_id': '/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '-1', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '0', 'world_size': '1'}
[2025-03-21 11:15:34,893] [13781] [281459877212512] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-03-21 11:15:34,895] [13781] [281459877212512] [llm] [INFO] [cpu_binding.py-238] : process 13781, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], cpu count 16
[2025-03-21 11:15:35,469] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : model_runner.quantize: w8a8sc, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16
[2025-03-21 11:15:40,902] [13781] [281459877212512] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-03-21 11:15:40,904] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<\uff5cbegin\u2581of\u2581sentence\uff5c>', 'eos_token': '<\uff5cend\u2581of\u2581sentence\uff5c>', 'pad_token': '<\uff5cend\u2581of\u2581sentence\uff5c>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<\uff5cend\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<\uff5cUser\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<\uff5cAssistant\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<\uff5cbegin\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib64/python3.11/logging/__init__.py", line 1113, in emit
stream.write(msg + self.terminator)
UnicodeEncodeError: 'ascii' codec can't encode character '\uff5c' in position 325: ordinal not in range(128)
Call stack:
[2025-03-21 11:15:40,920] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
File "/usr/lib64/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/usr/lib64/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/utils/log/multiprocess_logging_handler.py", line 80, in receive
self.log_handler.emit(record)
Message: 'init tokenizer done: LlamaTokenizerFast(name_or_path=\'/data/drx/DeepSeek-R1-Distill-Qwen-1.5B_out_w8a8sc\', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side=\'left\', truncation_side=\'right\', special_tokens={\'bos_token\': \'<\uff5cbegin\u2581of\u2581sentence\uff5c>\', \'eos_token\': \'<\uff5cend\u2581of\u2581sentence\uff5c>\', \'pad_token\': \'<\uff5cend\u2581of\u2581sentence\uff5c>\'}, clean_up_tokenization_spaces=False), added_tokens_decoder={\n\t151643: AddedToken("<\uff5cend\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151644: AddedToken("<\uff5cUser\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151645: AddedToken("<\uff5cAssistant\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151646: AddedToken("<\uff5cbegin\u2581of\u2581sentence\uff5c>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n\t151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),\n}'
Arguments: ()
[2025-03-21 11:15:40,996] [13781] [281459877212512] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-03-21 11:15:45,513] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8SparseCompressedLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8SparseCompressedLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8SparseCompressedLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorHead(
(linear): FastLinear()
)
)
[2025-03-21 11:15:45,571] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,571] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,572] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,572] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,573] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,573] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,574] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,574] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,575] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,576] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,576] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,577] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,578] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,579] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,580] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,581] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,582] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,582] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,583] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,583] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,584] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,585] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,585] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,586] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,586] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,587] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,588] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,589] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,590] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,591] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,592] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,593] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,593] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,594] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,595] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,596] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,597] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,598] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,598] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,599] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,599] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,600] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,601] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,602] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,603] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,603] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,604] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,604] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,605] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,605] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,606] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,606] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,607] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,607] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,608] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,608] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,609] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,609] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,610] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,610] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,611] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,611] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,612] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,612] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,613] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,613] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,614] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,614] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,615] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,616] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,617] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,617] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,618] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,618] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,619] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,619] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,620] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,620] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,621] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,622] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,622] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,623] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,624] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,625] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,626] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,627] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,628] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,628] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,629] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 2
[2025-03-21 11:15:45,629] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:45,632] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : trans to 29
[2025-03-21 11:15:46,184] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-03-21 11:15:46,197] [13781] [281459877212512] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 16, 128, 16])
[2025-03-21 11:15:46,199] [13781] [281459877212512] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281466043764272 id of vcache is 281466043764176
2025-03-21 11:15:48,541 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-21-11:15:48 (PID:13781, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 13781] 2025-03-21-11:15:48.341.339 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=994, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[13781]:
EXCEPTION STREAM:
Exception info:TGID=1727582, model id=65535, stream id=2, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=14, stream_id=2
Other info[0]:time=2025-03-21-11:15:48.257.677, function=process_hwts_error_exception, line=1320, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-03-21-11:15:48 (PID:13781, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 13781] 2025-03-21-11:15:48.341.339 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=994, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
DEVICE[0] PID[13781]:
EXCEPTION STREAM:
Exception info:TGID=1727582, model id=65535, stream id=2, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: Aicore exception, slot_id=14, stream_id=2
Other info[0]:time=2025-03-21-11:15:48.257.677, function=process_hwts_error_exception, line=1320, error code=0x26
2025-03-21 11:15:48,543 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
[root@localhost bin]# ^C
[root@localhost bin]#