Fault Locating
HCCL sets up independent maintenance and test connections with adjacent ranks based on the existing communicator information to provide the capability of broadcasting single point of failures in the cluster. (Note: HCCL controls the number of maintenance and test connections and the size of communication data. You do not need to concern about the performance loss caused by these maintenance and test connections on communication connections.) In this way, the plog of any rank will contain information about the fault root rank. The following table lists the supported fault detection capabilities.
Priority |
Exception |
Status (run/plog) |
ExceptionType (debug/plog) |
Criterion |
|---|---|---|---|---|
1 |
Network issue |
ERROR CQE |
Error cqe Occurred |
The system polls RoCE driver retransmission threshold crossing events, and maps the remote IP address through the QPN. |
2 |
Process suspension |
STUCK |
Stuck Occurred |
The system polls the input and output counts of all operators at an interval of 1/3 of HCCL_EXEC_TIMEOUT to check whether the process is suspended. |
3 |
Process exit |
LOST |
Heartbeat Lost Occurred |
No heartbeat packet is received from the remote end within 30s. |
To limit printed events, the Cluster Exception ERROR logs show only the first three valid events, prioritizing ERROR CQE over STUCK and LOST. To check all heartbeat events, search for them in logs in the run directory.
- After detecting an exception event, HCCL spreads and forwards it in the cluster. HCCL receives the exception event and print it in a run log.
Log format: [HeartbeatAbnormal]local rank[IP/ID]: crimer rank[IP/ID] status[4 exception event type] by informer rank[IP/ID].
- HeartbeatAbnormal: heartbeat exception event
- local rank: current rank
- crimer rank: root rank
- status: exception type
- by informer rank: user who reported the cluster fault
You can combine the keywords HeartbeatAbnormal and status to search for the logs. For example:
[INFO] HCCL(686,python):2025-10-23-07:52:59.191.363 [heartbeat.cc:951] [8970][TaskExecStage][HeartbeatAbnormal]local rank [127.10.0.1/1]: crimer rank [127.10.0.2/2] status[LOST] by informer rank [127.10.0.3/3]
- If an operator execution error is reported and the task exception callback function is called to notify HCCL, HCCL infers the most possible cause of the single point of failure based on the received exception event and the timeout event configuration such as HCCL_EXEC_TIMEOUT, and records the cause in the error logs.
Log format: [TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID], Arrival Time: [Week Month Day Hour:Minute:Second Year], Discoverer:[IP/ID], ExceptionType: [Exception type], Possible Reason: Possible cause.
- [TaskExecStage][HeartbeatAbnormal]: The cluster fault occurs during operator execution and is a heartbeat exception event.
- Cluster Exception Location: Location where the cluster fault occurs.
- Arrival Time: Time when the cluster fault occurs.
- Discoverer: Rank where the cluster fault is detected.
- ExceptionType: Exception type of the cluster fault.
- Possible Reason: Possible cause of the cluster fault.
You can use the keyword HeartbeatAbnormal to search for the logs. For example:
[ERROR]HCCL(835695,all_reduce_test):2025-10-23-17:28:06.049.385[task_exception_handler.cc:610][835695][TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID]:[127.10.0.1/1], Arrival Time:[Thu Oct 23 17:25:58 2025], Discoverer:[127.10.0.1/2], ExceptionType:[Heartbeat Lost Occurred], Possible Reason:1. Process has exited, 2. Network Disconnected
If no exception event occurs after the timeout, the cluster behavior may be inconsistent. Check factors such as the script, version, and dataset first. If necessary, enable the HCCL_ENTRY_LOG_ENABLE environment variable to trace operator-level behaviors.
- If the training/inference task is stopped before Notify times out or the task exception mechanism does not call the callback function to notify HCCL promptly due to some reasons, HCCL does not print exception information. You can still locate the root rank based on the exception events generated in the run logs. Then, identify the exception events. Generally, the LOST/ERROR CQE event near the system suspension time is the cause of the system suspension. The STUCK event detection time is (1/3 to 2/3) × HCCL_EXEC_TIMEOUT.
- Network exceptions and process exiting may cause both LOST and ERROR CQE events. Locate the fault based on the specific heartbeat event details, for example, ranks at both ends report LOST events of each other.