[2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-43] : model_config {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'}
[2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item local_rank value is 0.
[2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item infer_mode value is standard.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item model_instance_type value is Standard.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item speculation_gamma value is 0.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item backend_type value is atb.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named plugin_params, use default value .
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item plugin_params value is .
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named tp, use default value 1.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item tp value is 1.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named pp, use default value 1.
[2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item pp value is 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named ep, use default value 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item ep value is 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named dp, use default value 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item dp value is 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named es, use default value 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item es value is 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named sp, use default value 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item sp value is 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named cp, use default value 1.
[2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item cp value is 1.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item model_id value is /opt/zr_offline_packages/models/Qwen3-32B-w8a8/.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item local_rank value is 0.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item rank value is 0.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item globalRankIds value is .
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item globalWorldSize value is 0.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item world_size value is 2.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item npu_device_id value is 0.
[2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-135] : The list item npu_device_ids value is ['0', '1'].
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item cpu_mem value is 0.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item npu_mem value is -1.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_seq_len value is 2560.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_iter_times value is 512.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_prefill_tokens value is 8192.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item block_size value is 128.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item distributed_enable value is false.
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-54] : >>model_config after initialize {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'}
[2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [request_router.py-55] : init base config
[2026-05-06 10:39:26,309] [797] [281464188891552] [llm] [INFO] [metrics.py-56] : profiling is disenabled.
[2026-05-06 10:39:26,309] [797] [281464188891552] [llm] [INFO] [router_impl.py-167] : global rank id 0 get model config: {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'}
2026-05-06 10:39:27.821435 1079 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg
LogLevelDynamicHandler start
2026-05-06 10:39:27.821840 1068 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg
LogLevelDynamicHandler start
[2026-05-06 10:39:27.888905] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:27.888968] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:27.889466] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:27.889531] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:27.889714] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:27.890364] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter.
[2026-05-06 10:39:30,502] [800] [281458964296096] [llmmodels] [INFO] [cpu_binding.py-258] : rank_id: 1, device_id: 1, numa_id: 1, shard_devices: [0, 1], cpus: [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
[2026-05-06 10:39:30,504] [800] [281458964296096] [llmmodels] [INFO] [cpu_binding.py-287] : process 800, new_affinity is [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], cpu count 16
[2026-05-06 10:39:30,526] [797] [281464188891552] [llmmodels] [INFO] [cpu_binding.py-258] : rank_id: 0, device_id: 0, numa_id: 1, shard_devices: [0, 1], cpus: [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
[2026-05-06 10:39:30,528] [797] [281464188891552] [llmmodels] [INFO] [cpu_binding.py-287] : process 797, new_affinity is [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], cpu count 16
[2026-05-06 10:39:31,026] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-166] : model_runner.config: Qwen3Config(max_position_embeddings=40960, vocab_size=151936, quantize='w8a8_pdmix', rope_scaling=RopeScaling(factor=1.0, type='linear', long_factor=None, short_factor=None, low_freq_factor=None, high_freq_factor=None, original_max_position_embeddings=None, rope_type=None, rope_theta=None, attention_factor=None, beta_fast=32, beta_slow=1), quantization_config=QuantizationConfig(group_size=0, kv_quant_type=None, fa_quant_type=None, reduce_quant_type=None), reasoning_config=ReasoningConfig(start_reasoning_token_id=151667, end_reasoning_token_id=151668), generation_config=GenerationConfig(max_new_tokens=None, pad_token_id=151643, eos_token_id=[151645, 151643], bot_token_id=None, user_token_id=None, assistant_token_id=None, system_token_id=None, max_length=None), is_reasoning_model=True, hidden_size=5120, intermediate_size=25600, num_hidden_layers=64, num_attention_heads=64, num_key_value_heads=8, hidden_act='silu', initializer_range=0.02, rms_norm_eps=1e-06, use_cache=True, tie_word_embeddings=False, rope_theta=1000000, use_sliding_window=False, sliding_window=None, max_window_layers=64, attention_dropout=0.0, pdmix=False, attention_bias=False, use_qk_norm=True)
[2026-05-06 10:39:31,027] [797] [281464188891552] [llmmodels] [INFO] [dist.py-77] : Current process is setting device npu:0 for rank 0.
[2026-05-06 10:39:31,094] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-166] : model_runner.config: Qwen3Config(max_position_embeddings=40960, vocab_size=151936, quantize='w8a8_pdmix', rope_scaling=RopeScaling(factor=1.0, type='linear', long_factor=None, short_factor=None, low_freq_factor=None, high_freq_factor=None, original_max_position_embeddings=None, rope_type=None, rope_theta=None, attention_factor=None, beta_fast=32, beta_slow=1), quantization_config=QuantizationConfig(group_size=0, kv_quant_type=None, fa_quant_type=None, reduce_quant_type=None), reasoning_config=ReasoningConfig(start_reasoning_token_id=151667, end_reasoning_token_id=151668), generation_config=GenerationConfig(max_new_tokens=None, pad_token_id=151643, eos_token_id=[151645, 151643], bot_token_id=None, user_token_id=None, assistant_token_id=None, system_token_id=None, max_length=None), is_reasoning_model=True, hidden_size=5120, intermediate_size=25600, num_hidden_layers=64, num_attention_heads=64, num_key_value_heads=8, hidden_act='silu', initializer_range=0.02, rms_norm_eps=1e-06, use_cache=True, tie_word_embeddings=False, rope_theta=1000000, use_sliding_window=False, sliding_window=None, max_window_layers=64, attention_dropout=0.0, pdmix=False, attention_bias=False, use_qk_norm=True)
[2026-05-06 10:39:31,096] [800] [281458964296096] [llmmodels] [INFO] [dist.py-77] : Current process is setting device npu:1 for rank 1.
[2026-05-06 10:39:35,779] [797] [281464188891552] [llmmodels] [INFO] [dist.py-97] : Current process finished setting device npu:0 for rank 0.
[2026-05-06 10:39:35,780] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-199] : init tokenizer done
[2026-05-06 10:39:36,107] [800] [281458964296096] [llmmodels] [INFO] [dist.py-97] : Current process finished setting device npu:1 for rank 1.
[2026-05-06 10:39:36,470] [797] [281464188891552] [llmmodels] [INFO] [flash_causal_qwen2.py-150] : >>>> qwen_QwenDecoderModel is called.
[2026-05-06 10:39:36,817] [800] [281458964296096] [llmmodels] [INFO] [flash_causal_qwen2.py-150] : >>>> qwen_QwenDecoderModel is called.
Loading selected layers: 100%|██████████| 64/64 [00:03<00:00, 17.05layer/s]
[2026-05-06 10:39:42.682343] [797] [1079] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map
[2026-05-06 10:39:42,698] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-256] : Initialized model: FlashQwen2ForCausalLM
[2026-05-06 10:39:42,711] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-279] : Start transferring model to device npu:0
[2026-05-06 10:39:43.556481] [800] [1068] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map
[2026-05-06 10:39:43,573] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-256] : Initialized model: FlashQwen2ForCausalLM
[2026-05-06 10:39:43,586] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-279] : Start transferring model to device npu:1
[2026-05-06 10:39:46,933] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-281] : Model successfully transferred to device npu:0
[2026-05-06 10:39:47,018] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-321] : Successfully loaded model: FlashQwen2ForCausalLM
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0): FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(1-2): 2 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(3-5): 3 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(6-7): 2 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(8-10): 3 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(11): FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(12-42): 31 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(43-45): 3 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(46-61): 16 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(62): FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
(63): FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
(q_norm): QwenRMSNorm()
(k_norm): QwenRMSNorm()
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): W8A8PDMixLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): W8A8PDMixLinear()
)
)
(ln_1): QwenRMSNormBias()
(ln_2): QwenRMSNormBias()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2026-05-06 10:39:47,019] [797] [281464188891552] [llm] [INFO] [generator.py-253] : The effective eos_token_id is `[151645, 151643]`.
[2026-05-06 10:39:47,019] [797] [281464188891552] [llm] [INFO] [generator.py-261] : The effective pad_token_id is `151936`.
[2026-05-06 10:39:47,020] [797] [281464188891552] [llm] [INFO] [generator.py-269] : The effective bos_token_id is `151643`.
2026-05-06 10:39:47.319555 1079 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llm] [loglevel] [file:line] [status code] msg
LogLevelDynamicHandler start
[2026-05-06 10:39:47.319923] [797] [1079] [llm] [INFO] [post_processing_manager.cpp:170] Get post processing manager
[2026-05-06 10:39:47.319972] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470701662624
[2026-05-06 10:39:47.320006] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470693208480
[2026-05-06 10:39:47.320034] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470684754336
[2026-05-06 10:39:47.320059] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470676300192
[2026-05-06 10:39:47.320090] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470627606944
[2026-05-06 10:39:47.320118] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468309533088
[2026-05-06 10:39:47.320146] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468301078944
[2026-05-06 10:39:47.320173] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468292624800
[2026-05-06 10:39:47,321] [797] [281464188891552] [llm] [INFO] [cache_manager.py-92] : Before prefill warmup, peak mem: 17.99G, total_mem: 43.24G
[2026-05-06 10:39:47,321] [797] [281464188891552] [llm] [INFO] [generator.py-677] : warmup params: max_prefill_tokens=8192, max_seq_len=2560, max_input_len=2048, max_iter_times=512
[2026-05-06 10:39:47,322] [797] [281464188891552] [llm] [INFO] [generator.py-876] : `standard blocks` during warmup needs npu memory(GB): 0.015625
[2026-05-06 10:39:47,323] [797] [281464188891552] [llm] [INFO] [cache_pool.py-179] : Allocating npu_cache, acl format is NZ, sepd mode is off.
[2026-05-06 10:39:47,401] [797] [281464188891552] [llm] [ERROR] [logging.py-55] : Error in forward_tensor: 'FlashQwenLayer' object has no attribute 'attn'
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/status.py", line 17, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 85, in do_inference
self.initialize(execute_request.config)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 70, in initialize
initialize_result = self.router_impl.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 168, in initialize
self.generator = Generator(model_config={**model_config.model_config})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 291, in __init__
self.cache_manager = self.warm_up(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 485, in warm_up
npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 791, in __warmup_standard
npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 779, in __warmup_prefill
npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 880, in __auto_warmup
self.__execute_warm_up(cache_manager, input_metadata, dummy=True)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 657, in __execute_warm_up
self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode,
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 627, in _warm_up
super()._warm_up(model_inputs, **kwargs)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 270, in _warm_up
logits = self.forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 68, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 212, in forward
logits = self._forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 750, in _forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 133, in forward
result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 223, in forward_from_model_inputs
result = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 298, in forward_tensor
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 283, in forward_tensor
result = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 344, in forward
res = self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 519, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 197, in init_ascend_weight
decode_weight_wrapper = self.get_weights(quantize_type=QuantType.W8A8)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 176, in get_weights
weight_wrapper.register_layer(layer, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 316, in register_layer
self.register_layer_attn(layer, self.attn_wrapper, quantize_type)
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 214, in register_layer_attn
wrapper_module = get_module(layer, wrapper.wrapper_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 13, in get_module
obj = getattr(obj, name)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'FlashQwenLayer' object has no attribute 'attn'
[2026-05-06 10:39:47,409] [797] [281464188891552] [llm] [ERROR] [logging.py-55] : Core thread encountered an exception and will exit the process.: tid=281464188891552, tname=inference
[2026-05-06 10:39:47.437363] [718] [795] [server] [WARN] [llm_daemon.cpp:228] [MIE04W01011A] [daemon] Parent process [718] received exit signal[17], Thread 281459954067296
[2026-05-06 10:39:47.437446] [718] [795] [server] [WARN] [llm_daemon.cpp:234] [MIE04W01011A] [daemon] Child process [797] that sent the signal has exited
[2026-05-06 10:39:47.437450] [718] [795] [server] [INFO] [llm_daemon.cpp:240] [daemon] Process 797 exited normally with status 1
[2026-05-06 10:39:47.437463] [718] [795] [server] [WARN] [llm_daemon.cpp:272] [MIE04W01011A] [daemon] Successfully handled SIGCHLD, now killing process group
Daemon is killing, please wait about 15 seconds...
点赞
收藏
回复
分享
举报
已解决
环境背景
描述
期望：在Qwen3-32B模型上进行量化，模式：w8a8；并通过MindIE进行加载（参考：Qwen3-14B、Qwen3-32B pdmix W8A8量化章节）
异常完整信息
[2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-43] : model_config {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'} [2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item local_rank value is 0. [2026-05-06 10:39:26,299] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item infer_mode value is standard. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item model_instance_type value is Standard. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item speculation_gamma value is 0. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item backend_type value is atb. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named plugin_params, use default value . [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item plugin_params value is . [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named tp, use default value 1. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item tp value is 1. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named pp, use default value 1. [2026-05-06 10:39:26,300] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item pp value is 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named ep, use default value 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item ep value is 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named dp, use default value 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item dp value is 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named es, use default value 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item es value is 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named sp, use default value 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item sp value is 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-113] : There is no item named cp, use default value 1. [2026-05-06 10:39:26,301] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item cp value is 1. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item model_id value is /opt/zr_offline_packages/models/Qwen3-32B-w8a8/. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item local_rank value is 0. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item rank value is 0. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item globalRankIds value is . [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item globalWorldSize value is 0. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item world_size value is 2. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item npu_device_id value is 0. [2026-05-06 10:39:26,307] [797] [281464188891552] [llm] [INFO] [config.py-135] : The list item npu_device_ids value is ['0', '1']. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item cpu_mem value is 0. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item npu_mem value is -1. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_seq_len value is 2560. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_iter_times value is 512. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item max_prefill_tokens value is 8192. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item block_size value is 128. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-123] : The item distributed_enable value is false. [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [config.py-54] : >>model_config after initialize {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'} [2026-05-06 10:39:26,308] [797] [281464188891552] [llm] [INFO] [request_router.py-55] : init base config [2026-05-06 10:39:26,309] [797] [281464188891552] [llm] [INFO] [metrics.py-56] : profiling is disenabled. [2026-05-06 10:39:26,309] [797] [281464188891552] [llm] [INFO] [router_impl.py-167] : global rank id 0 get model config: {'kv_pool_async_write': 'false', 'async_inference': 'false', 'globalWorldSize': '0', 'kv_trans_timeout': '10', 'interNodeTlsPkPwd': 'security/grpc/pass/mindie_server_key_pwd.txt', 'max_iter_times': '512', 'block_size': '128', 'model_name': 'Qwen3-32B-w8a8', 'interNodeTlsCrlPath': 'security/grpc/certs/', 'interNodeTlsCert': 'security/grpc/certs/server.pem', 'asyncBatchscheduler': 'false', 'executor_type': 'LLM_EXECUTOR_PYTHON', 'kv_pool_backend': '', 'local_world_size': '2', 'speculation_gamma': '0', 'model_id': '/opt/zr_offline_packages/models/Qwen3-32B-w8a8/', 'interNodeTlsCaFiles': 'ca.pem,', 'interNodeKmcKsfMaster': 'tools/pmt/master/ksfa', 'deploy_type': 'INTER_PROCESS', 'interNodeTlsPk': 'security/grpc/keys/server.key.pem', 'interNodeTLSEnabled': '1', 'interNodeTlsCaPath': 'security/grpc/ca/', 'world_size': '2', 'kv_link_timeout': '1080', 'model_instance_type': 'Standard', 'max_batch_size': '200', 'slaveIPs': '', 'model_instance_number': '1', 'trust_remote_code': '0', 'globalRankIds': '', 'rank': '0', 'max_seq_len': '2560', 'interNodeKmcKsfStandby': 'tools/pmt/standby/ksfb', 'max_prefill_tokens': '8192', 'backend_bin_path': '/bin/', 'npu_device_id': '0', 'distributed_enable': 'false', 'interNodeTlsCrlFiles': 'server_crl.pem,', 'local_rank': '0', 'isMaster': '0', 'masterIP': '', 'max_input_len': '2048', 'backend_type': 'atb', 'async_scheduler_wait_time': '120', 'multiNodesInferEnabled': '0', 'kv_pool_config_path': '', 'infer_mode': 'standard', 'max_lora_rank': '0', 'max_loras': '0', 'npu_mem': '-1', 'localIP': '', 'multiNodesInferPort': '1120', 'npu_device_ids': '0,1', 'is_dmi_infer': '0', 'threadNum': '1', 'cpu_mem': '0'} 2026-05-06 10:39:27.821435 1079 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg LogLevelDynamicHandler start 2026-05-06 10:39:27.821840 1068 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg LogLevelDynamicHandler start [2026-05-06 10:39:27.888905] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:27.888968] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:27.889466] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:27.889531] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:27.889714] [797] [1079] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:27.890364] [800] [1068] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter. [2026-05-06 10:39:30,502] [800] [281458964296096] [llmmodels] [INFO] [cpu_binding.py-258] : rank_id: 1, device_id: 1, numa_id: 1, shard_devices: [0, 1], cpus: [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] [2026-05-06 10:39:30,504] [800] [281458964296096] [llmmodels] [INFO] [cpu_binding.py-287] : process 800, new_affinity is [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], cpu count 16 [2026-05-06 10:39:30,526] [797] [281464188891552] [llmmodels] [INFO] [cpu_binding.py-258] : rank_id: 0, device_id: 0, numa_id: 1, shard_devices: [0, 1], cpus: [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] [2026-05-06 10:39:30,528] [797] [281464188891552] [llmmodels] [INFO] [cpu_binding.py-287] : process 797, new_affinity is [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], cpu count 16 [2026-05-06 10:39:31,026] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-166] : model_runner.config: Qwen3Config(max_position_embeddings=40960, vocab_size=151936, quantize='w8a8_pdmix', rope_scaling=RopeScaling(factor=1.0, type='linear', long_factor=None, short_factor=None, low_freq_factor=None, high_freq_factor=None, original_max_position_embeddings=None, rope_type=None, rope_theta=None, attention_factor=None, beta_fast=32, beta_slow=1), quantization_config=QuantizationConfig(group_size=0, kv_quant_type=None, fa_quant_type=None, reduce_quant_type=None), reasoning_config=ReasoningConfig(start_reasoning_token_id=151667, end_reasoning_token_id=151668), generation_config=GenerationConfig(max_new_tokens=None, pad_token_id=151643, eos_token_id=[151645, 151643], bot_token_id=None, user_token_id=None, assistant_token_id=None, system_token_id=None, max_length=None), is_reasoning_model=True, hidden_size=5120, intermediate_size=25600, num_hidden_layers=64, num_attention_heads=64, num_key_value_heads=8, hidden_act='silu', initializer_range=0.02, rms_norm_eps=1e-06, use_cache=True, tie_word_embeddings=False, rope_theta=1000000, use_sliding_window=False, sliding_window=None, max_window_layers=64, attention_dropout=0.0, pdmix=False, attention_bias=False, use_qk_norm=True) [2026-05-06 10:39:31,027] [797] [281464188891552] [llmmodels] [INFO] [dist.py-77] : Current process is setting device npu:0 for rank 0. [2026-05-06 10:39:31,094] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-166] : model_runner.config: Qwen3Config(max_position_embeddings=40960, vocab_size=151936, quantize='w8a8_pdmix', rope_scaling=RopeScaling(factor=1.0, type='linear', long_factor=None, short_factor=None, low_freq_factor=None, high_freq_factor=None, original_max_position_embeddings=None, rope_type=None, rope_theta=None, attention_factor=None, beta_fast=32, beta_slow=1), quantization_config=QuantizationConfig(group_size=0, kv_quant_type=None, fa_quant_type=None, reduce_quant_type=None), reasoning_config=ReasoningConfig(start_reasoning_token_id=151667, end_reasoning_token_id=151668), generation_config=GenerationConfig(max_new_tokens=None, pad_token_id=151643, eos_token_id=[151645, 151643], bot_token_id=None, user_token_id=None, assistant_token_id=None, system_token_id=None, max_length=None), is_reasoning_model=True, hidden_size=5120, intermediate_size=25600, num_hidden_layers=64, num_attention_heads=64, num_key_value_heads=8, hidden_act='silu', initializer_range=0.02, rms_norm_eps=1e-06, use_cache=True, tie_word_embeddings=False, rope_theta=1000000, use_sliding_window=False, sliding_window=None, max_window_layers=64, attention_dropout=0.0, pdmix=False, attention_bias=False, use_qk_norm=True) [2026-05-06 10:39:31,096] [800] [281458964296096] [llmmodels] [INFO] [dist.py-77] : Current process is setting device npu:1 for rank 1. [2026-05-06 10:39:35,779] [797] [281464188891552] [llmmodels] [INFO] [dist.py-97] : Current process finished setting device npu:0 for rank 0. [2026-05-06 10:39:35,780] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-199] : init tokenizer done [2026-05-06 10:39:36,107] [800] [281458964296096] [llmmodels] [INFO] [dist.py-97] : Current process finished setting device npu:1 for rank 1. [2026-05-06 10:39:36,470] [797] [281464188891552] [llmmodels] [INFO] [flash_causal_qwen2.py-150] : >>>> qwen_QwenDecoderModel is called. [2026-05-06 10:39:36,817] [800] [281458964296096] [llmmodels] [INFO] [flash_causal_qwen2.py-150] : >>>> qwen_QwenDecoderModel is called. Loading selected layers: 100%|██████████| 64/64 [00:03<00:00, 17.05layer/s] [2026-05-06 10:39:42.682343] [797] [1079] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map [2026-05-06 10:39:42,698] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-256] : Initialized model: FlashQwen2ForCausalLM [2026-05-06 10:39:42,711] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-279] : Start transferring model to device npu:0 [2026-05-06 10:39:43.556481] [800] [1068] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map [2026-05-06 10:39:43,573] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-256] : Initialized model: FlashQwen2ForCausalLM [2026-05-06 10:39:43,586] [800] [281458964296096] [llmmodels] [INFO] [model_runner.py-279] : Start transferring model to device npu:1 [2026-05-06 10:39:46,933] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-281] : Model successfully transferred to device npu:0 [2026-05-06 10:39:47,018] [797] [281464188891552] [llmmodels] [INFO] [model_runner.py-321] : Successfully loaded model: FlashQwen2ForCausalLM FlashQwen2ForCausalLM( (rotary_embedding): PositionRotaryEmbedding() (attn_mask): AttentionMask() (transformer): FlashQwenModel( (wte): TensorParallelEmbedding() (h): ModuleList( (0): FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (1-2): 2 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (3-5): 3 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (6-7): 2 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (8-10): 3 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (11): FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (12-42): 31 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (43-45): 3 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (46-61): 16 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (62): FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) (63): FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) (q_norm): QwenRMSNorm() (k_norm): QwenRMSNorm() ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): W8A8PDMixLinear() ) (c_proj): TensorParallelRowLinear( (linear): W8A8PDMixLinear() ) ) (ln_1): QwenRMSNormBias() (ln_2): QwenRMSNormBias() ) ) (ln_f): QwenRMSNorm() ) (lm_head): TensorParallelHead( (linear): FastLinear() ) ) [2026-05-06 10:39:47,019] [797] [281464188891552] [llm] [INFO] [generator.py-253] : The effective eos_token_id is `[151645, 151643]`. [2026-05-06 10:39:47,019] [797] [281464188891552] [llm] [INFO] [generator.py-261] : The effective pad_token_id is `151936`. [2026-05-06 10:39:47,020] [797] [281464188891552] [llm] [INFO] [generator.py-269] : The effective bos_token_id is `151643`. 2026-05-06 10:39:47.319555 1079 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu][processid] [threadid] [llm] [loglevel] [file:line] [status code] msg LogLevelDynamicHandler start [2026-05-06 10:39:47.319923] [797] [1079] [llm] [INFO] [post_processing_manager.cpp:170] Get post processing manager [2026-05-06 10:39:47.319972] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470701662624 [2026-05-06 10:39:47.320006] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470693208480 [2026-05-06 10:39:47.320034] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470684754336 [2026-05-06 10:39:47.320059] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470676300192 [2026-05-06 10:39:47.320090] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281470627606944 [2026-05-06 10:39:47.320118] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468309533088 [2026-05-06 10:39:47.320146] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468301078944 [2026-05-06 10:39:47.320173] [797] [1079] [llm] [INFO] [thread_pool.cpp:54] Create pthread id: 281468292624800 [2026-05-06 10:39:47,321] [797] [281464188891552] [llm] [INFO] [cache_manager.py-92] : Before prefill warmup, peak mem: 17.99G, total_mem: 43.24G [2026-05-06 10:39:47,321] [797] [281464188891552] [llm] [INFO] [generator.py-677] : warmup params: max_prefill_tokens=8192, max_seq_len=2560, max_input_len=2048, max_iter_times=512 [2026-05-06 10:39:47,322] [797] [281464188891552] [llm] [INFO] [generator.py-876] : `standard blocks` during warmup needs npu memory(GB): 0.015625 [2026-05-06 10:39:47,323] [797] [281464188891552] [llm] [INFO] [cache_pool.py-179] : Allocating npu_cache, acl format is NZ, sepd mode is off. [2026-05-06 10:39:47,401] [797] [281464188891552] [llm] [ERROR] [logging.py-55] : Error in forward_tensor: 'FlashQwenLayer' object has no attribute 'attn' Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/status.py", line 17, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 85, in do_inference self.initialize(execute_request.config) File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/request_router.py", line 70, in initialize initialize_result = self.router_impl.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/connector/request_router/router_impl.py", line 168, in initialize self.generator = Generator(model_config={**model_config.model_config}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 291, in __init__ self.cache_manager = self.warm_up( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 485, in warm_up npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 791, in __warmup_standard npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 779, in __warmup_prefill npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 880, in __auto_warmup self.__execute_warm_up(cache_manager, input_metadata, dummy=True) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 657, in __execute_warm_up self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode, File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 627, in _warm_up super()._warm_up(model_inputs, **kwargs) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 270, in _warm_up logits = self.forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 68, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 212, in forward logits = self._forward(model_inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 750, in _forward logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 133, in forward result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 223, in forward_from_model_inputs result = self.forward_tensor( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 298, in forward_tensor raise e File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 283, in forward_tensor result = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 344, in forward res = self.model.forward(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 519, in forward self.init_ascend_weight() File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 197, in init_ascend_weight decode_weight_wrapper = self.get_weights(quantize_type=QuantType.W8A8) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 176, in get_weights weight_wrapper.register_layer(layer, quantize_type) File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 316, in register_layer self.register_layer_attn(layer, self.attn_wrapper, quantize_type) File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 214, in register_layer_attn wrapper_module = get_module(layer, wrapper.wrapper_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 13, in get_module obj = getattr(obj, name) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1695, in __getattr__ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'") AttributeError: 'FlashQwenLayer' object has no attribute 'attn' [2026-05-06 10:39:47,409] [797] [281464188891552] [llm] [ERROR] [logging.py-55] : Core thread encountered an exception and will exit the process.: tid=281464188891552, tname=inference [2026-05-06 10:39:47.437363] [718] [795] [server] [WARN] [llm_daemon.cpp:228] [MIE04W01011A] [daemon] Parent process [718] received exit signal[17], Thread 281459954067296 [2026-05-06 10:39:47.437446] [718] [795] [server] [WARN] [llm_daemon.cpp:234] [MIE04W01011A] [daemon] Child process [797] that sent the signal has exited [2026-05-06 10:39:47.437450] [718] [795] [server] [INFO] [llm_daemon.cpp:240] [daemon] Process 797 exited normally with status 1 [2026-05-06 10:39:47.437463] [718] [795] [server] [WARN] [llm_daemon.cpp:272] [MIE04W01011A] [daemon] Successfully handled SIGCHLD, now killing process group Daemon is killing, please wait about 15 seconds...疑惑