root@davinci-mini:/usr/local/Ascend/mindie-llm# torchrun --nproc_per_node 2          --master_port 20037          -m examples.run_pa          --model_path /home/ai/DeepSeek-R1-Distill-Qwen-1.5B/          --input_texts 'What is deep learning?'          --max_output_length 20 

[2025-02-21 04:51:06,240] torch.distributed.run: [WARNING] 

[2025-02-21 04:51:06,240] torch.distributed.run: [WARNING] ***************************************** 

[2025-02-21 04:51:06,240] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 

[2025-02-21 04:51:06,240] torch.distributed.run: [WARNING] ***************************************** 

Traceback (most recent call last): 

  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 

    Traceback (most recent call last): 

return _run_code(code, main_globals, None, 

  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 

  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 

    exec(code, run_globals) 

  File "/usr/local/Ascend/mindie-llm/examples/run_pa.py", line 509, in <module> 

    return _run_code(code, main_globals, None, 

  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 

    exec(code, run_globals) 

  File "/usr/local/Ascend/mindie-llm/examples/run_pa.py", line 509, in <module> 

    pa_runner = PARunner(**input_dict) 

  File "/usr/local/Ascend/mindie-llm/examples/run_pa.py", line 74, in __init__ 

    self.model = ModelRunner( 

  File "/usr/local/Ascend/mindie-llm/atb_llm/runner/model_runner.py", line 85, in __init__ 

    pa_runner = PARunner(**input_dict) 

  File "/usr/local/Ascend/mindie-llm/examples/run_pa.py", line 74, in __init__ 

    self.model = ModelRunner( 

  File "/usr/local/Ascend/mindie-llm/atb_llm/runner/model_runner.py", line 85, in __init__ 

        load_atb_speed() 

  File "/usr/local/Ascend/mindie-llm/atb_llm/utils/initial.py", line 53, in load_atb_speed 

load_atb_speed() 

  File "/usr/local/Ascend/mindie-llm/atb_llm/utils/initial.py", line 53, in load_atb_speed 

    torch.classes.load_library(lib_path) 

  File "/usr/local/lib/python3.10/dist-packages/torch/_classes.py", line 51, in load_library 

    torch.classes.load_library(lib_path) 

  File "/usr/local/lib/python3.10/dist-packages/torch/_classes.py", line 51, in load_library 

    torch.ops.load_library(path) 

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 852, in load_library 

    torch.ops.load_library(path) 

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 852, in load_library 

        ctypes.CDLL(path)ctypes.CDLL(path) 

 

  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__ 

  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__ 

        self._handle = _dlopen(self._name, mode)self._handle = _dlopen(self._name, mode) 

 

OSError: OSError/usr/local/Ascend/mindie-llm/lib/libatb_speed_operations.so: undefined symbol: aclnnMoeInitRoutingV2GetWorkspaceSize: 

/usr/local/Ascend/mindie-llm/lib/libatb_speed_operations.so: undefined symbol: aclnnMoeInitRoutingV2GetWorkspaceSize