Fault Locating
HCCL sets up independent maintenance and test connections with adjacent ranks based on the existing communicator information to provide the capability of broadcasting single point of failures in the cluster. (Note: HCCL controls the number of maintenance and test connections and the size of communication data. You do not need to concern about the performance loss caused by these maintenance and test connections on communication connections.) In this way, the plog of any rank will contain information about the fault root rank. The following table lists the supported fault detection capabilities.
Priority |
Exception |
Status (run/plog) |
ExceptionType (debug/plog) |
Criterion |
|---|---|---|---|---|
1 |
Network issue |
ERROR CQE |
Error cqe Occurred |
The event indicating that RoCE driver retransmission exceeds the limit is polled periodically, and the remote IP address is mapped through the QPN. |
2 |
Process suspension |
STUCK |
Stuck Occurred |
The number of input and output times of all operators is polled at an interval of 1/3 of HCCL_EXEC_TIMEOUT to check whether the process is suspended. |
3 |
Process exit |
LOST |
Heartbeat Lost Occurred |
No heartbeat packet is received from the remote end within 30s. |
To control the number of printed events, only the first three valid events are printed in the Cluster Exception ERROR logs, and the priority is as follows: ERROR CQE > STUCK > LOST. To confirm all heartbeat events, search for them in logs in the run directory.
- HCCL detects an exception event and spreads the information across the cluster. It logs the exception events during the operation.
Log format: [HeartbeatAbnormal]local rank[IP/ID]: crimer rank[IP/ID] status [4 exception event type]: crimer rank[IP/ID] status [4 exception event type] by informer rank[IP/ID].
- HeartbeatAbnormal: heartbeat exception event
- local rank: current node
- crimer rank: root node
- status: exception type
- by by informer rank: user who reported the cluster fault
You can combine the keywords HeartbeatAbnormal and status to search for the logs. For example:
[INFO] HCCL(686,python):2025-10-23-07:52:59.191.363 [heartbeat.cc:951] [8970][TaskExecStage][HeartbeatAbnormal]local rank [127.10.0.1/1]: crimer rank [127.10.0.2/2] status[LOST] by informer rank [127.10.0.3/3]
- If an operator execution error is reported and the task exception callback function is called to notify HCCL, HCCL infers the most possible cause of the single point of failure based on the received exception event and the timeout event configuration such as HCCL_EXEC_TIMEOUT, and records the cause in the error logs.
Log format: [TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID], Arrival Time: [Week Month Day Hour:Minute:Second Year], Discoverer:[IP/ID], ExceptionType: [Exception type], Possible Reason: Possible cause.
- [TaskExecStage][HeartbeatAbnormal]: The cluster fault occurs during operator execution and is a heartbeat exception event.
- Cluster Exception Location: Location where the cluster fault occurs.
- Arrival Time: Time when the cluster fault occurs.
- Discoverer: Node where the cluster fault is detected.
- ExceptionType: Exception type of the cluster fault.
- Possible Reason: Possible cause of the cluster fault.
You can use the keyword HeartbeatAbnormal to search for the logs. For example:
[ERROR]HCCL(835695,all_reduce_test):2025-10-23-17:28:06.049.385[task_exception_handler.cc:610][835695][TaskExecStage][HeartbeatAbnormal]Cluster Exception Location[IP/ID]:[127.10.0.1/1], Arrival Time:[Thu Oct 23 17:25:58 2025], Discoverer:[127.10.0.1/2], ExceptionType:[Heartbeat Lost Occurred], Possible Reason:1. Process has exited, 2. Network Disconnected
If no exception event occurs after the timeout, the cluster behavior may be inconsistent. In this case, check factors such as the script, version, and dataset first. If necessary, enable the HCCL_ENTRY_LOG_ENABLE environment variable to trace operator-level behavior.
- If the training/inference task is stopped before Notify times out or the task exception mechanism does not call the callback function to notify HCCL in a timely manner due to some reasons, HCCL does not print exception information. You can still locate the root rank based on the exception events generated during system running in the run logs. In this case, you need to identify exception events. Generally, the LOST/ERROR CQE event near the system suspension time is the cause of the system suspension. The STUCK event detection time is (1/3 to 2/3) × HCCL_EXEC_TIMEOUT.
- Network exceptions and process exiting may cause both LOST and ERROR CQE events. Locate the fault based on the specific heartbeat event details, for example, ranks at both ends report LOST events of each other.