Example: Process Suspension or Remote End Heartbeat Loss

Symptom

The CANN logs contain the keyword "Cluster Exception Location," as shown below:

Remote end heartbeat loss:

[ERROR]HCCL(835695,all_reduce_test):2025-10-23-17:28:06.049.385[task_exception_handler.cc:610][835695][TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID]:[127.10.0.1/1], Arrival Time:[Thu Oct 23 17:25:58 2025], Discoverer:[127.10.0.1/2], ExceptionType:[Heartbeat Lost Occurred], Possible Reason:1. Process has exited, 2. Network Disconnected

Process suspension:

[ERROR]HCCL(1219039,all_reduce_test):2025-10-23-21:05:09.859.568[task_exception_handler.cc:610] [1219039][TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID]:[127.10.0.1/1], Arrival Time:[Thu Oct 23 21:03:19 2025], ExceptionType:[Stuck Occurred], Possible Reason:1.Host process is stuck, 2.Device task is stuck

Possible Causes and Fault Locating

You can identify the exception type and the rank where the exception occurs from the error logs.

• Cluster Exception Location: rank where the exception occurs.
• Arrival Time: time when the exception is broadcast to the local end.
• ExceptionType: exception type, including heartbeat loss (Heartbeat Lost Occurred), process suspension (Stuck Occurred), and network packet loss (Error CQE Occurred).
• Possible Reason: possible causes and troubleshooting methods of the exception.
  • Heartbeat Lost Occurred: Check whether the rank where the exception occurs exits before the exception is broadcast to the local end or whether the network between ranks is abnormal and cannot be connected.
  • Stuck Occurred: Check whether the service process of the rank where the exception occurs is suspended on a rank or a deadlock occurs.
  • Error cqe Occurred: Check whether a CQE error occurs on the rank where the exception occurs.