[root@user bin]# ./mindieservice_daemon
2025-04-11 06:12:54,294 [INFO] config.py:30 - model_config {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '1', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '1024', 'max_iter_times': '512', 'max_prefill_tokens': '4096', 'max_seq_len': '1024', 'model_cut_policy': 'standard', 'model_id': '/models', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '2', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '1', 'world_size': '1'}
2025-04-11 06:12:54,294 [INFO] config.py:35 - do not init dmi config
2025-04-11 06:12:54,294 [INFO] standard_model.py:145 - rank id 0 get model config: {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '1', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '1024', 'max_iter_times': '512', 'max_prefill_tokens': '4096', 'max_seq_len': '1024', 'model_cut_policy': 'standard', 'model_id': '/models', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '2', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '1', 'world_size': '1'}
[2025-04-11 06:12:58,020] [3084] [281466953580896] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[2025-04-11 06:12:58,022] [3084] [281466953580896] [llm] [INFO] [cpu_binding.py-238] : process 3084, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7], cpu count 8
[2025-04-11 06:12:58,531] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16
[2025-04-11 06:13:04,024] [3084] [281466953580896] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-04-11 06:13:04,025] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : init tokenizer done: Qwen2TokenizerFast(name_or_path='/models', vocab_size=151643, model_max_length=131072, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151646: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151648: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151649: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-04-11 06:13:04,030] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
[2025-04-11 06:13:04,051] [3084] [281466953580896] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-04-11 06:13:05,171] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-23): 24 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-04-11 06:13:05,298] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,299] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,299] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,300] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,300] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,301] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,302] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,302] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,303] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,303] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,304] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,304] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,305] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,305] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,306] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,307] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,307] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,309] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,311] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,311] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,312] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,312] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,313] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,313] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,314] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,314] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,315] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,315] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,316] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,316] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,317] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,317] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,318] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,318] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,319] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,319] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,320] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,320] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,321] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,321] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,322] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,322] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,323] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,323] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,324] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,324] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,325] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,325] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,326] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,326] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,327] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,327] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,328] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,328] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,330] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,330] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,331] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,331] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,332] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,332] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,334] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,336] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,336] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,337] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,337] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,338] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,338] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,339] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,339] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,340] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,340] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,341] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,341] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,342] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,342] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,343] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,343] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,344] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,344] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,345] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,345] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,347] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[W compiler_depend.ts:387] Warning: EZ3002: [PID: 3084] 2025-04-11-06:13:05.460.141 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
(function ExecFunc)
[2025-04-11 06:13:05,978] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1365, 8, 128, 16])
2025-04-11 06:13:05,979 [ERROR] model.py:39 - [Model] >>> Exception:The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Identity.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-11-06:13:05 (PID:3084, Device:0, RankID:-1) ERR00100 PTA call acl api failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 356, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 348, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 454, in forward
self.init_kvcache(kv_cache)
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 392, in init_kvcache
k_caches = [torch_npu.npu_format_cast_(k_cache, 29) for k_cache in k_caches]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 392, in <listcomp>
k_caches = [torch_npu.npu_format_cast_(k_cache, 29) for k_cache in k_caches]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_ops.py", line 692, in __call__
return self._op(*args, **kwargs or {})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Identity.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-11-06:13:05 (PID:3084, Device:0, RankID:-1) ERR00100 PTA call acl api failed
2025-04-11 06:13:05,984 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
环境是docker的mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts bash,宿主机也报这个错误
点赞
收藏
回复
分享
举报
已解决
2025-04-11 06:12:54,294 [INFO] config.py:30 - model_config {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '1', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '1024', 'max_iter_times': '512', 'max_prefill_tokens': '4096', 'max_seq_len': '1024', 'model_cut_policy': 'standard', 'model_id': '/models', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '2', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '1', 'world_size': '1'}
2025-04-11 06:12:54,294 [INFO] config.py:35 - do not init dmi config
2025-04-11 06:12:54,294 [INFO] standard_model.py:145 - rank id 0 get model config: {'backend_bin_path': '/usr/local/Ascend/mindie/1.0.0/mindie-llm/bin/', 'backend_log_file': '/usr/local/Ascend/mindie/1.0.0/mindie-service/logs/mindservice.log', 'backend_modelInstance_id': '0', 'backend_type': 'atb', 'block_size': '128', 'cpu_mem': '1', 'deploy_type': 'INTER_PROCESS', 'dp': '0', 'ep': '0', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'globalRankIds': '', 'globalWorldSize': '0', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'interNodeTLSEnabled': '1', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeTlsCaPath': 'security/grpc/ca/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'isMaster': '0', 'localIP': '', 'local_rank': '0', 'log_error': '1', 'log_file_num': '20', 'log_file_size': '20', 'log_info': '1', 'log_verbose': '0', 'log_warning': '1', 'masterIP': '', 'max_input_len': '1024', 'max_iter_times': '512', 'max_prefill_tokens': '4096', 'max_seq_len': '1024', 'model_cut_policy': 'standard', 'model_id': '/models', 'model_instance_number': '1', 'model_instance_type': 'Standard', 'model_name': 'qwen2', 'multiNodesInferEnabled': '0', 'multiNodesInferPort': '1120', 'npu_device_id': '0', 'npu_device_ids': '0', 'npu_mem': '2', 'pp': '0', 'rank': '0', 'slaveIPs': '', 'speculation_gamma': '0', 'tp': '0', 'trust_remote_code': '1', 'world_size': '1'}
[2025-04-11 06:12:58,020] [3084] [281466953580896] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
[2025-04-11 06:12:58,022] [3084] [281466953580896] [llm] [INFO] [cpu_binding.py-238] : process 3084, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7], cpu count 8
[2025-04-11 06:12:58,531] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16
[2025-04-11 06:13:04,024] [3084] [281466953580896] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-04-11 06:13:04,025] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : init tokenizer done: Qwen2TokenizerFast(name_or_path='/models', vocab_size=151643, model_max_length=131072, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>']}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151646: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151648: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151649: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-04-11 06:13:04,030] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
[2025-04-11 06:13:04,051] [3084] [281466953580896] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-04-11 06:13:05,171] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-23): 24 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-04-11 06:13:05,298] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,299] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,299] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,300] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,300] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,301] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,302] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,302] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,303] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,303] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,304] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,304] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,305] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,305] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,306] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,307] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,307] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,308] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,309] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,310] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,311] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,311] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,312] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,312] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,313] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,313] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,314] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,314] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,315] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,315] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,316] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,316] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,317] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,317] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,318] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,318] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,319] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,319] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,320] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,320] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,321] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,321] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,322] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,322] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,323] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,323] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,324] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,324] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,325] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,325] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,326] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,326] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,327] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,327] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,328] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,328] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,329] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,330] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,330] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,331] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,331] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,332] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,332] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,333] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,334] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,335] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,336] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,336] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,337] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,337] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,338] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,338] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,339] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,339] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,340] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,340] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,341] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,341] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,342] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,342] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,343] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,343] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,344] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,344] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,345] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,345] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-04-11 06:13:05,347] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : trans to 29
[W compiler_depend.ts:387] Warning: EZ3002: [PID: 3084] 2025-04-11-06:13:05.460.141 Optype [TransData] of Ops kernel [AIcoreEngine] is unsupported. Reason: [tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[tbe-custom]:op type TransData is not found in this op store.[tbe-custom1]:op type TransData is not found in this op store.[Dynamic shape check]: The format and dtype is not precisely equivalent to format and dtype in op information library.
Possible Cause: The operator type is unsupported in the operator information library due to specification mismatch.
Solution: Submit an issue to request for support at https://gitee.com/ascend, or remove this type of operators from your model.
TraceBack (most recent call last):
Optype [TransData] of Ops kernel [aicpu_ascend_kernel] is unsupported. Reason: Transdata op, groups should be greater than 1, but now is 1.
No supported Ops kernel and engine are found for [trans_TransData_0], optype [TransData].
Assert ((SelectEngine(node_ptr, exclude_engines, is_check_support_success, op_info)) == ge::SUCCESS) failed[FUNC:operator()][FILE:engine_place.cc][LINE:148]
RunAllSubgraphs failed, graph=online.[FUNC:RunAllSubgraphs][FILE:engine_place.cc][LINE:122]
build graph failed, graph id:0, ret:4294967295[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1618]
[Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
[Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
(function ExecFunc)
[2025-04-11 06:13:05,978] [3084] [281466953580896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1365, 8, 128, 16])
2025-04-11 06:13:05,979 [ERROR] model.py:39 - [Model] >>> Exception:The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Identity.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-11-06:13:05 (PID:3084, Device:0, RankID:-1) ERR00100 PTA call acl api failed
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 356, in warm_up
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 348, in warm_up
self._generate_inputs_warm_up_backend(input_metadata, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 378, in _generate_inputs_warm_up_backend
self.generator_backend.warm_up(model_inputs, inference_mode=inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 198, in warm_up
super().warm_up(model_inputs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 170, in warm_up
_ = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 38, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 153, in forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 89, in forward
logits = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 116, in forward_tensor
logits = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 454, in forward
self.init_kvcache(kv_cache)
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 392, in init_kvcache
k_caches = [torch_npu.npu_format_cast_(k_cache, 29) for k_cache in k_caches]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 392, in <listcomp>
k_caches = [torch_npu.npu_format_cast_(k_cache, 29) for k_cache in k_caches]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/_ops.py", line 692, in __call__
return self._op(*args, **kwargs or {})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is Identity.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2025-04-11-06:13:05 (PID:3084, Device:0, RankID:-1) ERR00100 PTA call acl api failed
2025-04-11 06:13:05,984 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'}
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Daemon is killing...
Killed
环境是docker的mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts bash,宿主机也报这个错误