Atlas 800 训练服务器 （型号 9000）
OS CentOS 7.6
驱动：23.0.0
固件：7.1.0.3.220
CANN：8.0.RC3.alpha001
Toolkit：8.0.RC3.alpha001
按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-32B#atlas-800i-a2-w8a8%E9%87%8F%E5%8C%96中的步骤进行部署，到“执行对话测试”一步执行torchrun --nproc_per_node 4 \ --master_port 20037 \ -m examples.run_pa \ --model_path /home/DeepSeek-R1-Distill-Qwen-32B \ --max_output_length 20 代码后报错，请问怎么解决,报错如下：
[root@localhost atb-models]# torchrun --nproc_per_node 4 \
--master_port 20037 \
-m examples.run_pa \
--model_path /home/DeepSeek-R1-Distill-Qwen-32B \
--max_output_length 20
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING]
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] *****************************************
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] *****************************************
[2025-02-10 10:56:58,736] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : Skip binding cpu.
[2025-02-10 10:56:59,695] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16
[2025-02-10 10:57:05,001] [9527] [281473644185888] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,068] [9527] [281473644185888] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,167] [9529] [281473789413664] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,234] [9529] [281473789413664] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,528] [9526] [281473258047776] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,531] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/home/DeepSeek-R1-Distill-Qwen-32B', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<｜begin▁of▁sentence｜>', 'eos_token': '<｜end▁of▁sentence｜>', 'pad_token': '<｜end▁of▁sentence｜>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<｜end▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<｜User｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<｜Assistant｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<｜begin▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-02-10 10:57:05,542] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False)
[2025-02-10 10:57:05,583] [9526] [281473258047776] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,684] [9528] [281473233602848] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,749] [9528] [281473233602848] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:16,432] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-63): 64 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-02-10 10:57:17,630] [9527] [281473644185888] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9527] 2025-02-10-10:57:17.691.995 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:17 (PID:9527, Device:1, RankID:1) ERR01100 OPS call acl api failed
[2025-02-10 10:57:17,867] [9529] [281473789413664] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9529] 2025-02-10-10:57:17.928.348 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:17 (PID:9529, Device:3, RankID:3) ERR01100 OPS call acl api failed
[2025-02-10 10:57:17,983] [9528] [281473233602848] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9528] 2025-02-10-10:57:18.044.310 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:18 (PID:9528, Device:2, RankID:2) ERR01100 OPS call acl api failed
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : hbm_capacity(GB): 13.2353515625, init_memory(GB): 1.1911816401407123
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : pa_runner: PARunner(model_path=/home/DeepSeek-R1-Distill-Qwen-32B, input_text=None, max_position_embeddings=None, max_input_length=1024, max_output_length=20, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.bfloat16, block_size=128, model_config=ModelConfig(num_heads=10, num_kv_heads=2, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=64, device=npu:0, dtype=torch.bfloat16, soc_info=NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=4, rank=0, pp_rank=0, pp_groups=[[0], [1], [2], [3]], micro_batch_size=1) , cla_share_factor=1, model_type=qwen2), max_memory=14211350528,
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : ---------------begin warm_up---------------
[2025-02-10 10:57:18,100] [9526] [281473258047776] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
[2025-02-10 10:57:18,104] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : ------total req num: 1, infer start--------
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9526] 2025-02-10-10:57:18.163.175 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:18 (PID:9526, Device:0, RankID:0) ERR01100 OPS call acl api failed
[2025-02-10 10:57:39,190] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 9526) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_pa FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 9527)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 9528)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 9529)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 9526)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
点赞
收藏
回复
分享
举报
已解决
Atlas 800 训练服务器 （型号 9000）
OS CentOS 7.6
驱动：23.0.0
固件：7.1.0.3.220
CANN：8.0.RC3.alpha001
Toolkit：8.0.RC3.alpha001
按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-32B#atlas-800i-a2-w8a8%E9%87%8F%E5%8C%96中的步骤进行部署，到“执行对话测试”一步执行torchrun --nproc_per_node 4 \ --master_port 20037 \ -m examples.run_pa \ --model_path /home/DeepSeek-R1-Distill-Qwen-32B \ --max_output_length 20 代码后报错，请问怎么解决,报错如下：
[root@localhost atb-models]# torchrun --nproc_per_node 4 \
--master_port 20037 \
-m examples.run_pa \
--model_path /home/DeepSeek-R1-Distill-Qwen-32B \
--max_output_length 20
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING]
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] *****************************************
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2025-02-10 10:56:49,177] torch.distributed.run: [WARNING] *****************************************
[2025-02-10 10:56:58,736] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : Skip binding cpu.
[2025-02-10 10:56:59,695] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16
[2025-02-10 10:57:05,001] [9527] [281473644185888] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,068] [9527] [281473644185888] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,167] [9529] [281473789413664] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,234] [9529] [281473789413664] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,528] [9526] [281473258047776] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,531] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/home/DeepSeek-R1-Distill-Qwen-32B', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<｜begin▁of▁sentence｜>', 'eos_token': '<｜end▁of▁sentence｜>', 'pad_token': '<｜end▁of▁sentence｜>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={
151643: AddedToken("<｜end▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<｜User｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151645: AddedToken("<｜Assistant｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151646: AddedToken("<｜begin▁of▁sentence｜>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
[2025-02-10 10:57:05,542] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False)
[2025-02-10 10:57:05,583] [9526] [281473258047776] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:05,684] [9528] [281473233602848] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set
[2025-02-10 10:57:05,749] [9528] [281473233602848] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called.
[2025-02-10 10:57:16,432] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : model:
FlashQwen2ForCausalLM(
(rotary_embedding): PositionRotaryEmbedding()
(attn_mask): AttentionMask()
(transformer): FlashQwenModel(
(wte): TensorParallelEmbedding()
(h): ModuleList(
(0-63): 64 x FlashQwenLayer(
(attn): FlashQwenAttention(
(rotary_emb): PositionRotaryEmbedding()
(c_attn): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(mlp): QwenMLP(
(act): SiLU()
(w2_w1): TensorParallelColumnLinear(
(linear): FastLinear()
)
(c_proj): TensorParallelRowLinear(
(linear): FastLinear()
)
)
(ln_1): QwenRMSNorm()
(ln_2): QwenRMSNorm()
)
)
(ln_f): QwenRMSNorm()
)
(lm_head): TensorParallelHead(
(linear): FastLinear()
)
)
[2025-02-10 10:57:17,630] [9527] [281473644185888] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9527] 2025-02-10-10:57:17.691.995 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:17 (PID:9527, Device:1, RankID:1) ERR01100 OPS call acl api failed
[2025-02-10 10:57:17,867] [9529] [281473789413664] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9529] 2025-02-10-10:57:17.928.348 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:17 (PID:9529, Device:3, RankID:3) ERR01100 OPS call acl api failed
[2025-02-10 10:57:17,983] [9528] [281473233602848] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9528] 2025-02-10-10:57:18.044.310 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:18 (PID:9528, Device:2, RankID:2) ERR01100 OPS call acl api failed
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : hbm_capacity(GB): 13.2353515625, init_memory(GB): 1.1911816401407123
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : pa_runner: PARunner(model_path=/home/DeepSeek-R1-Distill-Qwen-32B, input_text=None, max_position_embeddings=None, max_input_length=1024, max_output_length=20, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.bfloat16, block_size=128, model_config=ModelConfig(num_heads=10, num_kv_heads=2, num_kv_heads_origin=8, head_size=128, k_head_size=128, v_head_size=128, num_layers=64, device=npu:0, dtype=torch.bfloat16, soc_info=NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=4, rank=0, pp_rank=0, pp_groups=[[0], [1], [2], [3]], micro_batch_size=1) , cla_share_factor=1, model_type=qwen2), max_memory=14211350528,
[2025-02-10 10:57:18,099] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : ---------------begin warm_up---------------
[2025-02-10 10:57:18,100] [9526] [281473258047776] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0703125GB memory
[2025-02-10 10:57:18,104] [9526] [281473258047776] [llm] [INFO] [logging.py-331] : ------total req num: 1, infer start--------
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module>
pa_runner.warm_up()
File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up
generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req
req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking
req_finished = generate_token(model, cache_manager, input_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token
logits = model.forward(
^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward
return self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward
self.init_ascend_weight()
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight
weight_wrapper = self.get_weights()
^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights
weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__
self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error!
EZ9999: [PID: 9526] 2025-02-10-10:57:18.163.175 Parse dynamic kernel config fail.
TraceBack (most recent call last):
AclOpKernelInit failed opType
ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed.
[ERROR] 2025-02-10-10:57:18 (PID:9526, Device:0, RankID:0) ERR01100 OPS call acl api failed
[2025-02-10 10:57:39,190] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 9526) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
examples.run_pa FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 9527)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 9528)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 9529)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2025-02-10_10:57:39
host : localhost.localdomain
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 9526)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================