Fault Locating

After the communicator is initialized and the parameter plane connection is set up, HCCL orchestrates and dispatches communication operator tasks. For communication operator task orchestration, the notify synchronization mechanism is used before data communication to ensure that the remote end is ready to receive data from the local end. If the process is suspended or exits, a network fault occurs, or the dispatched communication operators are inconsistent for some ranks due to an exception, execution waiting will time out on most ranks. If such a problem occurs, locate the fault point first. The following figure shows the overall fault locating process.

Figure 1 Fault locating process in the task dispatch and execution phase

• HCCL has a cluster heartbeat mechanism. When an exception is detected on a rank, the exception is broadcast to each rank in the cluster through this mechanism. You can search the CANN logs of any rank for heartbeat exception event information. For details about the mechanism and log information, see Cluster Heartbeat Mechanism.
• If no heartbeat exception event information is found, check whether there are cluster behavior inconsistency problems based on the task exception information. For details, see Task Exception Mechanism.