运行命令为：torchrun --nproc_per_node 1 --master_port 20037 -m examples.run_pa --model_path /weights/deepseek-r1-7b/ --max_output_length 20

报错：

ImportError: /usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib/libatb.so: undefined symbol: HcclAllGatherV

完整报错：

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 10, in <module>

    from atb_llm.runner.model_runner import ModelRunner

  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 12, in <module>

    import atb_llm.nn as atb_llm_nn

  File "/usr/local/Ascend/atb-models/atb_llm/nn/__init__.py", line 3, in <module>

    from atb_llm.nn.modules import (

  File "/usr/local/Ascend/atb-models/atb_llm/nn/modules/__init__.py", line 6, in <module>

    from .normalization import RmsNorm as RmsNorm

  File "/usr/local/Ascend/atb-models/atb_llm/nn/modules/normalization.py", line 5, in <module>

    from ..network import Tensor, Node, get_default_net

  File "/usr/local/Ascend/atb-models/atb_llm/nn/network.py", line 11, in <module>

    import _libatb_torch as atb

ImportError: /usr/local/Ascend/nnal/atb/latest/atb/cxx_abi_0/lib/libatb.so: undefined symbol: HcclAllGatherV

[ERROR] 2025-08-13-10:27:14 (PID:163, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception

[2025-08-13 10:27:17,835] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 163) of binary: /usr/bin/python3

Traceback (most recent call last):

  File "/usr/local/bin/torchrun", line 8, in <module>

    sys.exit(main())

             ^^^^^^

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper

    return f(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 806, in main

    run(args)

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/run.py", line 797, in run

    elastic_launch(

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__

    return launch_agent(self._config, self._entrypoint, list(args))

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib64/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent

    raise ChildFailedError(

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

============================================================

examples.run_pa FAILED

------------------------------------------------------------

Failures:

  <NO_OTHER_FAILURES>

------------------------------------------------------------

Root Cause (first observed failure):

[0]:

  time      : 2025-08-13_10:27:17

  host      : huawei-NucBox-2

  rank      : 0 (local_rank: 0)

  exitcode  : 1 (pid: 163)

  error_file: <N/A>

  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

============================================================