RuntimeError: Preprocess failed before run graph 1.
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- Framework Error Message: (For framework developers)
----------------------------------------------------
Distribute Task Failed,
error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/AllGather-op54301_3692653
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
---
点赞
收藏
回复
分享
举报
RuntimeError: Preprocess failed before run graph 1.
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
----------------------------------------------------
- Framework Error Message: (For framework developers)
----------------------------------------------------
Distribute Task Failed,
error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/AllGather-op54301_3692653
----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
EI9999: Inner Error!
EI9999 connected p2p timeout, timeout:600 s.local logicDevid:0,remote physic id:7 The possible causes are as follows:1.the connection between this device and the target device is abnormal 2.an exception occurred at the target devices 3.The ranktable is not matched.[FUNC:WaitP2PConnected][FILE:p2p_mgmt.cc][LINE:275]
TraceBack (most recent call last):
(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
---