镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-3
openeuler24.03
docker run -itd --net=host --shm-size=32g --privileged --name 7b_test --device=/dev/davinci0 --device=/dev/davinci2 --device=/dev/davinci_manager -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro -v /models/model/deepseek-ai/DeepSeek-R1-Distill-QWen-7B/:/model -e ASCEND_VISIBLE_DEVICES=0,2 swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts bash
一张卡可以跑起来，两张卡就报错
torchrun --nproc_per_node 2 --master_port 20030 -m examples.run_pa --model_path /model --input_texts "简单介绍一下你自己" --max_output_length 104
报错：
1 | tee run.log torchrun --nproc_per_node 2 --master_port 20030 -m examples.run_pa --model_path /model --input_texts "简单介绍一下你自己" --max_output_length 104 2>&1
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING]
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] *****************************************
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloa ded, please further tune the variable for optimal performance in your application as needed.
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] *****************************************
[2025-05-13 14:24:37,255] [5503] [281473033550400] [llm] [INFO] [cpu_binding.py-212] : rank_id: 1, device_id: 1, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-05-13 14:24:37,257] [5503] [281473033550400] [llm] [INFO] [cpu_binding.py-238] : process 5503, new_affinity is [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 3 0, 31], cpu count 16
[2025-05-13 14:24:37,265] [5502] [281473120344640] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-05-13 14:24:37,267] [5502] [281473120344640] [llm] [INFO] [cpu_binding.py-238] : process 5502, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], cp u count 16
[2025-05-13 14:24:38,079] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: N one, model_runner.dtype: torch.float16
[2025-05-13 14:24:44,684] [5502] [281473120344640] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-05-13 14:24:44,685] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/model', vocab_size=151643, model_max_ length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<｜begin▁of▁sentence｜>', 'eos_token': '<｜end▁of▁sentence｜>', 'pad_to ken': '<｜end▁of▁sentence｜>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<｜end▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<｜User｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<｜Assistant｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<｜begin▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-05-13 14:24:44,691] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
[2025-05-13 14:24:44,708] [5503] [281473033550400] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-05-13 14:24:44,761] [5502] [281473120344640] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-05-13 14:24:44,780] [5503] [281473033550400] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 489, in <module>
pa_runner = PARunner(**input_dict)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 85, in __init__
self.model.load_weights(**kw_args)
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 149, in load_weights
self.model = self.model_cls(self.config, weights, inference_mode=self.inference_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 24, in __init__
super().__init__(config, weights, **kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 115, in __init__
self.cu_seqlen_tensor_fake = torch.tensor([0], dtype=torch.int).to(self.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: ACL stream synchronize failed, error code:507015
[ERROR] 2025-05-13-14:24:56 (PID:5503, Device:1, RankID:-1) ERR99999 UNKNOWN application exception
[2025-05-13 14:25:01,255] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-05-13 14:25:03,288] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : hbm_capacity(GB): 28.4130859375, init_memory(GB): 9.376318358816206
[2025-05-13 14:25:03,288] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : pa_runner: PARunner(model_path=/model, input_text=None, max_position_embeddings=None, max_in put_length=1024, max_output_length=104, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=14, num_kv_heads=2, num_kv_heads_origin=4, head_size=128, k_head_size=128, v_head_size=128, num_layers=28, device=npu:0, dt ype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, , max_memory=30508318720,
[2025-05-13 14:25:03,290] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : ---------------begin warm_up---------------
[2025-05-13 14:25:03,290] [5502] [281473120344640] [llm] [INFO] [cache.py-79] : kv cache will allocate 0.03076171875GB memory
[2025-05-13 14:25:03,293] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,293] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,294] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,294] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,295] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,295] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,296] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,297] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,297] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,299] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,299] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,300] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,300] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,301] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,301] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,303] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,303] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,305] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,305] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,306] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,306] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,307] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,307] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,309] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,309] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,310] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,310] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,311] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,311] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,312] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,312] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,313] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,313] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,314] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,314] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,315] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,315] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,316] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,316] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,317] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,317] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,319] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,319] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,320] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,320] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,321] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,321] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,322] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,322] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,323] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,323] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,324] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,324] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,326] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,326] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,327] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,327] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,329] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,329] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,331] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,331] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,332] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,332] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,333] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,333] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,335] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,335] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,336] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,336] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,337] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,337] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,338] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,338] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,339] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,339] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,341] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,341] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,342] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,342] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,343] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,343] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,344] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,347] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:04,008] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([9, 16, 128, 16])
[2025-05-13 14:25:04,015] [5502] [281473120344640] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([9, 16, 128, 16])
[2025-05-13 14:25:04,016] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281470063082096 id of vcache is 281470063083440
[2025-05-13 14:25:05,653] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 5502 closing signal SIGTERM
[2025-05-13 14:25:06,017] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 5503) of binary: /usr/bin/python3
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_pa FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-05-13_14:25:05
host : linux
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 5503)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
驱动信息：
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.t29 Version: 24.1.t29 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310P3 | OK | NA 53 0 / 0 |
| 0 0 | 0000:01:00.0 | 0 1870 / 29095 |
+===============================+=================+======================================================+
| 64 310P3 | OK | NA 53 0 / 0 |
| 0 1 | 0000:02:00.0 | 0 1870 / 29095 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
[root@linux /]# ls /dev/davinci*
/dev/davinci0 /dev/davinci2 /dev/davinci_manager
点赞
收藏
回复
分享
举报
已解决
镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-3
openeuler24.03
docker run -itd --net=host --shm-size=32g --privileged --name 7b_test --device=/dev/davinci0 --device=/dev/davinci2 --device=/dev/davinci_manager -v /usr/local/Ascend/driver:/usr/local/Ascend/driver:ro -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi:ro -v /models/model/deepseek-ai/DeepSeek-R1-Distill-QWen-7B/:/model -e ASCEND_VISIBLE_DEVICES=0,2 swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts bash
一张卡可以跑起来，两张卡就报错
torchrun --nproc_per_node 2 --master_port 20030 -m examples.run_pa --model_path /model --input_texts "简单介绍一下你自己" --max_output_length 104
报错：
1 | tee run.log torchrun --nproc_per_node 2 --master_port 20030 -m examples.run_pa --model_path /model --input_texts "简单介绍一下你自己" --max_output_length 104 2>&1
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING]
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] *****************************************
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloa ded, please further tune the variable for optimal performance in your application as needed.
[2025-05-13 14:24:30,647] torch.distributed.run: [WARNING] *****************************************
[2025-05-13 14:24:37,255] [5503] [281473033550400] [llm] [INFO] [cpu_binding.py-212] : rank_id: 1, device_id: 1, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-05-13 14:24:37,257] [5503] [281473033550400] [llm] [INFO] [cpu_binding.py-238] : process 5503, new_affinity is [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 3 0, 31], cpu count 16
[2025-05-13 14:24:37,265] [5502] [281473120344640] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
[2025-05-13 14:24:37,267] [5502] [281473120344640] [llm] [INFO] [cpu_binding.py-238] : process 5502, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], cp u count 16
[2025-05-13 14:24:38,079] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: N one, model_runner.dtype: torch.float16
[2025-05-13 14:24:44,684] [5502] [281473120344640] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-05-13 14:24:44,685] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/model', vocab_size=151643, model_max_ length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<｜begin▁of▁sentence｜>', 'eos_token': '<｜end▁of▁sentence｜>', 'pad_to ken': '<｜end▁of▁sentence｜>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<｜end▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<｜User｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<｜Assistant｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<｜begin▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-05-13 14:24:44,691] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False)
[2025-05-13 14:24:44,708] [5503] [281473033550400] [llm] [INFO] [dist.py-77] : initialize_distributed has been Set
[2025-05-13 14:24:44,761] [5502] [281473120344640] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
[2025-05-13 14:24:44,780] [5503] [281473033550400] [llm] [INFO] [flash_causal_qwen2.py-115] : >>>> qwen_QwenDecoderModel is called.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 489, in <module>
pa_runner = PARunner(**input_dict)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 85, in __init__
self.model.load_weights(**kw_args)
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 149, in load_weights
self.model = self.model_cls(self.config, weights, inference_mode=self.inference_mode,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 24, in __init__
super().__init__(config, weights, **kwargs)
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 115, in __init__
self.cu_seqlen_tensor_fake = torch.tensor([0], dtype=torch.int).to(self.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: ACL stream synchronize failed, error code:507015
[ERROR] 2025-05-13-14:24:56 (PID:5503, Device:1, RankID:-1) ERR99999 UNKNOWN application exception
[2025-05-13 14:25:01,255] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-27): 28 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-05-13 14:25:03,288] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : hbm_capacity(GB): 28.4130859375, init_memory(GB): 9.376318358816206
[2025-05-13 14:25:03,288] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : pa_runner: PARunner(model_path=/model, input_text=None, max_position_embeddings=None, max_in put_length=1024, max_output_length=104, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=14, num_kv_heads=2, num_kv_heads_origin=4, head_size=128, k_head_size=128, v_head_size=128, num_layers=28, device=npu:0, dt ype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=202, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, , max_memory=30508318720,
[2025-05-13 14:25:03,290] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : ---------------begin warm_up---------------
[2025-05-13 14:25:03,290] [5502] [281473120344640] [llm] [INFO] [cache.py-79] : kv cache will allocate 0.03076171875GB memory
[2025-05-13 14:25:03,293] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,293] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,294] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,294] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,295] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,295] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,296] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,297] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,297] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,298] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,299] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,299] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,300] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,300] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,301] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,301] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,302] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,303] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,303] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,304] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,305] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,305] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,306] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,306] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,307] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,307] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,308] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,309] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,309] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,310] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,310] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,311] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,311] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,312] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,312] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,313] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,313] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,314] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,314] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,315] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,315] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,316] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,316] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,317] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,317] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,318] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,319] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,319] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,320] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,320] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,321] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,321] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,322] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,322] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,323] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,323] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,324] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,324] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,325] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,326] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,326] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,327] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,327] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,328] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,329] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,329] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,330] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,331] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,331] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,332] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,332] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,333] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,333] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,334] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,335] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,335] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,336] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,336] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,337] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,337] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,338] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,338] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,339] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,339] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,340] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,341] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,341] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,342] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,342] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,343] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,343] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,344] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:03,347] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : trans to 29
[2025-05-13 14:25:04,008] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([9, 16, 128, 16])
[2025-05-13 14:25:04,015] [5502] [281473120344640] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([9, 16, 128, 16])
[2025-05-13 14:25:04,016] [5502] [281473120344640] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281470063082096 id of vcache is 281470063083440
[2025-05-13 14:25:05,653] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 5502 closing signal SIGTERM
[2025-05-13 14:25:06,017] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 5503) of binary: /usr/bin/python3
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_pa FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-05-13_14:25:05
host : linux
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 5503)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
驱动信息：
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.t29 Version: 24.1.t29 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 0 310P3 | OK | NA 53 0 / 0 |
| 0 0 | 0000:01:00.0 | 0 1870 / 29095 |
+===============================+=================+======================================================+
| 64 310P3 | OK | NA 53 0 / 0 |
| 0 1 | 0000:02:00.0 | 0 1870 / 29095 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
[root@linux /]# ls /dev/davinci*
/dev/davinci0 /dev/davinci2 /dev/davinci_manager