底层报错

 

Traceback (most recent call last): 

  File "/home/ma-user/work/tsingge/packages/mindformers/research/qwen/run_qwen.py", line 165, in <module> 

    main(task=args.task, 

  File "/home/ma-user/work/tsingge/packages/mindformers/research/qwen/run_qwen.py", line 113, in main 

    result = task.predict(input_data=prompt, 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/_checkparam.py", line 1313, in wrapper 

    return func(*args, **kwargs) 

  File "/home/ma-user/work/tsingge/packages/mindformers/mindformers/trainer/trainer.py", line 640, in predict 

    output_result = self.trainer.predict( 

  File "/home/ma-user/work/tsingge/packages/mindformers/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 316, in predict 

    return self.predict_process(config=config, 

  File "/home/ma-user/work/tsingge/packages/mindformers/mindformers/trainer/base_trainer.py", line 875, in predict_process 

    transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) 

  File "/home/ma-user/work/tsingge/packages/mindformers/mindformers/trainer/utils.py", line 328, in transform_and_load_checkpoint 

    build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) 

  File "/home/ma-user/work/tsingge/packages/mindformers/mindformers/trainer/utils.py", line 450, in build_model 

    model.infer_predict_layout(dataset) 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 1882, in infer_predict_layout 

    predict_net.compile(*predict_data) 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 997, in compile 

    _cell_graph_executor.compile(self, phase=self.phase, 

  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 1547, in compile 

    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode()) 

RuntimeError: Preprocess failed before run graph 0. 

 

---------------------------------------------------- 

- Ascend Error Message: 

---------------------------------------------------- 

EI9999: Inner Error! 

EI9999  connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:4 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:274] 

        TraceBack (most recent call last): 

 

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) 

 

---------------------------------------------------- 

- Framework Error Message: (For framework developers) 

---------------------------------------------------- 

Distribute Task Failed,  

error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/AllGather-op9008_554315 

 

---------------------------------------------------- 

- Ascend Error Message: 

---------------------------------------------------- 

EI9999: Inner Error! 

EI9999  connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:4 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:274] 

        TraceBack (most recent call last): 

 

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) 

EI9999: Inner Error! 

EI9999  connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:4 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:274] 

        TraceBack (most recent call last): 

 

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description) 

 

---------------------------------------------------- 

- C++ Call Stack: (For framework developers) 

---------------------------------------------------- 

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_kernel_executor.cc:286 PreprocessBeforeRunGraph 

mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_runtime_core.cc:375 LoadTask 

mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:140 Distribute 

求救