欧拉系统 torchrun --nproc_per_node 2 --master_port 20030 -m examples.run_pa --model_path /model --input_texts "简单介绍一下你自己"  --max_output_length 102 

Traceback (most recent call last): 

  File "<frozen runpy>", line 198, in _run_module_as_main 

  File "<frozen runpy>", line 88, in _run_code 

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 491, in <module> 

    pa_runner.warm_up() 

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 215, in warm_up 

    self.model.forward( 

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward 

    return self.model.forward(**kwargs) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward 

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator 

    acl_model_out = model_operation.execute(acl_inputs, acl_param) 

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document. 

 

Traceback (most recent call last): 

  File "<frozen runpy>", line 198, in _run_module_as_main 

  File "<frozen runpy>", line 88, in _run_code 

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 491, in <module> 

    pa_runner.warm_up() 

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 215, in warm_up 

    self.model.forward( 

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 193, in forward 

    return self.model.forward(**kwargs) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 458, in forward 

    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill) 

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 374, in execute_ascend_operator 

    acl_model_out = model_operation.execute(acl_inputs, acl_param) 

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

RuntimeError: Execute fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to findthe first error. For more details, see the MindIE official document. 

 

[ERROR] 2025-05-20-12:01:01 (PID:177, Device:0, RankID:-1) ERR99999 UNKNOWN application exception 

[ERROR] 2025-05-20-12:01:01 (PID:178, Device:1, RankID:-1) ERR99999 UNKNOWN application exception 

[2025-05-20 12:01:08,120] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 177) of binary: /usr/bin/python3 

Traceback (most recent call last): 

  File "/usr/local/bin/torchrun", line 8, in <module> 

    sys.exit(main()) 

             ^^^^^^ 

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper 

    return f(*args, **kwargs) 

           ^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main 

    run(args) 

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run 

    elastic_launch( 

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__ 

    return launch_agent(self._config, self._entrypoint, list(args)) 

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent 

    raise ChildFailedError( 

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

============================================================ 

examples.run_pa FAILED 

------------------------------------------------------------ 

Failures: 

[1]: 

  time      : 2025-05-20_12:01:08 

  host      : linux 

  rank      : 1 (local_rank: 1) 

  exitcode  : 1 (pid: 178) 

  error_file: <N/A> 

  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html 

------------------------------------------------------------ 

Root Cause (first observed failure): 

[0]: 

  time      : 2025-05-20_12:01:08 

  host      : linux 

  rank      : 0 (local_rank: 0) 

  exitcode  : 1 (pid: 177) 

  error_file: <N/A> 

  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html 

============================================================ 

[root@linux /]#