Train epoch time: 3826.046 ms, per step time: 1913.023 ms

time: Wed Nov  1 21:56:44 2023 local_rank: 0, epoch: 0, step: 1422, loss is 3.5810506343841553, overflow is False, loss scale is 262144.0

Train epoch time: 3823.689 ms, per step time: 1911.844 ms

[WARNING] DEVICE(140901,fffd327cf120,python):2023-11-01-22:27:39.392.246 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:742] GetDumpPath] The environment variable 'MS_OM_PATH' is not set, the files of node dump will save to the process local path, as ./rank_id/node_dump/...

[ERROR] DEVICE(140901,fffd327cf120,python):2023-11-01-22:27:39.393.673 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:760] DumpTaskExceptionInfo] Task fail infos task_id: 4, stream_id: 102, tid: 140901, device_id: 0, retcode: 507011 ( model execute failed)

[WARNING] MD(140901,fffb11a20120,python):2023-11-01-22:27:39.571.365 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:280] SendDataToAscend] Thread has already been terminated.

Traceback (most recent call last):
...

RuntimeError: Run task for graph:kernel_graph_1 error! The details refer to 'Ascend Error Message'.

----------------------------------------------------

- Ascend Error Message:

----------------------------------------------------

EI0002: The wait execution of the Notify register times out. Reason: The Notify register has not received the Notify record from remote rank [3].base information: [streamID:[102], taskID[4], taskType[Notify Wait], tag[HcomAllGather_6629421139219749105_0].] task information: [notify id:[0x0000000000000018], stage:[ffffffff], remote rank:[3].

there are(is) 2 abnormal device(s):

  server:[192.168.246.223][4], server:[192.168.246.223][0], Heartbeat Lost Occurred, Possible Reason: 1. Process has exited, 2. Network Disconnected

  server:[192.168.246.223][2], server:[192.168.246.223][0], Heartbeat Lost Occurred, Possible Reason: 1. Process has exited, 2. Network Disconnected

]

        Possible Cause: 1. An exception occurs during the execution on some NPUs in the cluster. As a result, collective communication operation failed.2. The execution speed on some NPU in the cluster is too slow to complete a communication operation within the timeout interval. (default 1800s, You can set the interval by using HCCL_EXEC_TIMEOUT.)3. The number of training samples of each NPU is inconsistent.4. Packet loss or other connectivity problems occur on the communication link.

        Solution: 1. If this error is reported on part of these ranks, check other ranks to see whether other errors have been reported earlier.2. If this error is reported for all ranks, check whether the error reporting time is consistent (the maximum difference must not exceed 1800s). If not, locate the cause or adjust the locate the cause or set the HCCL_EXEC_TIMEOUT environment variable to a larger value.3. Check whether the completion queue element (CQE) of the error exists in the plog(grep -rn 'error cqe'). If so, check the network connection status. (For details, see the TLS command and HCCN connectivity check examples.)4. Ensure that the number of training samples of each NPU is consistent.

        TraceBack (most recent call last):

        Notify wait execute failed, device_id=0, stream_id=102, task_id=4, flip_num=0, notify_id=3[FUNC:GetError][FILE:stream.cc][LINE:1133]

        rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:49]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------

- C++ Call Stack: (For framework developers)

----------------------------------------------------

mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:256 RunGraph

如何解决这个问题。