昇腾论坛

服务器：1台、4块Atlas 300I DUO卡。镜像：mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts，模型DeepSeek-R1-Distill-Qwen-14B

NPU信息：

执行torchrun --nproc_per_node 2 --master_port 20037 -m examples.run_pa --model_path /home/data/DeepSeek-R1-Distill-Qwen-14B --max_output_length 20 无报错；

执行torchrun --nproc_per_node 4或6或8 --master_port 20037 -m examples.run_pa --model_path /home/data/DeepSeek-R1-Distill-Qwen-14B --max_output_length 20 报错如下：

Traceback (most recent call last):

[2025-02-28 16:43:40,241] [17409] [281461757266208] [llm] [INFO] [logging.py-331] : ------total req num: 1, infer start--------

File "<frozen runpy>", line 198, in _run_module_as_main

File "<frozen runpy>", line 88, in _run_code

File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 500, in <module>

generate_texts, token_nums, _ = pa_runner.infer(**infer_params)generate_texts, token_nums, _ = pa_runner.infer(**infer_params)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer

generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)

File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 135, in generate_req

req_finished = generate_token(model, cache_manager, batch)

^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 25, in generate_token

logits = model.forward(

^ ^ ^^^^^^^^^^^^^^^^^^^Traceback (most recent call last):

^^^^^

File "<frozen runpy>", line 198, in _run_module_as_main

^^ File "<frozen runpy>", line 88, in _run_code

File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 500, in <module>

File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

return self.model.forward(**kwargs)

^generate_texts, token_nums, _ = pa_runner.infer(**infer_params)

File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer

logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 135, in generate_req

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^req_finished = generate_token(model, cache_manager, batch)^^

^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ Traceback (most recent call last):

^^ ^

File "<frozen runpy>", line 198, in _run_module_as_main

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

File "<frozen runpy>", line 88, in _run_code

File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 500, in <module>

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 25, in generate_token

acl_model_out = model_operation.execute(acl_inputs, acl_param)acl_model_out = model_operation.execute(acl_inputs, acl_param)

logits = model.forward(

^ ^ ^ ^ generate_texts, token_nums, _ = pa_runner.infer(**infer_params) ^

^ ^ ^ ^ ^ ^ ^^^^^^ ^^^ ^

^ ^^ ^ File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^return self.model.forward(**kwargs) ^^

^^ ^^ ^^ ^^ ^^ ^^ ^^ ^^^ ^^^ ^^^ ^^^ ^^^ ^^^ ^^^ ^^^ ^

^ ^^^RuntimeError^RuntimeError^: ^: ^Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

^Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^ File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 284, in infer

^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager)

File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 135, in generate_req

logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

req_finished = generate_token(model, cache_manager, batch)

^ ^ ^ ^ ^ ^ ^ ^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^ File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 25, in generate_token

^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

logits = model.forward(

^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward

acl_model_out = model_operation.execute(acl_inputs, acl_param)

^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^return self.model.forward(**kwargs)^^

^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward

logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator

acl_model_out = model_operation.execute(acl_inputs, acl_param)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document.

[ERROR] 2025-02-28-16:43:51 (PID:17410, Device:1, RankID:-1) ERR99999 UNKNOWN application exception

[ERROR] 2025-02-28-16:43:51 (PID:17409, Device:0, RankID:-1) ERR99999 UNKNOWN application exception

[ERROR] 2025-02-28-16:43:51 (PID:17411, Device:2, RankID:-1) ERR99999 UNKNOWN application exception

[ERROR] 2025-02-28-16:43:51 (PID:17412, Device:3, RankID:-1) ERR99999 UNKNOWN application exception

[2025-02-28 16:44:01,378] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 17409) of binary: /usr/bin/python3

Traceback (most recent call last):

File "/usr/local/bin/torchrun", line 8, in <module>

sys.exit(main())

^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper

return f(*args, **kwargs)

^^^^^^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main

run(args)

File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run

elastic_launch(

File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__

return launch_agent(self._config, self._entrypoint, list(args))

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent

raise ChildFailedError(

torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

============================================================

examples.run_pa FAILED

------------------------------------------------------------

Failures:

[1]:

time : 2025-02-28_16:44:01

host : localhost.localdomain

rank : 1 (local_rank: 1)

exitcode : 1 (pid: 17410)

error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[2]:

time : 2025-02-28_16:44:01

host : localhost.localdomain

rank : 2 (local_rank: 2)

exitcode : 1 (pid: 17411)

error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[3]:

time : 2025-02-28_16:44:01

host : localhost.localdomain

rank : 3 (local_rank: 3)

exitcode : 1 (pid: 17412)

error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

------------------------------------------------------------

Root Cause (first observed failure):

[0]:

time : 2025-02-28_16:44:01

host : localhost.localdomain

rank : 0 (local_rank: 0)

exitcode : 1 (pid: 17409)

error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

===============================================

关于昇腾

新闻与活动

交流与资讯

支持与服务

开源社区