Fault Locating

When a communication operator is called, HCCL creates a socket connection through the parameter plane network over TCP to exchange addresses and other information needed for the service. If some ranks do not call the expected communication operator due to a fault, they cannot initiate the connection request. Also, if there are network issues and inconsistent behaviors, some ranks cannot respond to connection requests. In this case, other ranks will report socket connection timeouts.

Connection timeouts can spread through ranks because of the HCCL algorithm and operator call sequence. If a timeout happens, find the fault point first.

HCCL provides the following mechanism in the parameter plane connection setup phase to assist in quickly locating faults.

Mechanism for Positioning Connection Setup Root Rank

Connection setup issues can spread through ranks. For example, if the connection between rank 0 and rank 1 times out, and then the connection between rank 1 and rank 2 times out, rank 0 will report a connection timeout with rank 1. However, the real problem is the failed connection between rank 1 and rank 2. Therefore, it is difficult to locate the root rank where the connection setup fails in the cluster. HCCL immediately starts fault detection connection after connection setup fails on the service. The implementation principles are as follows:

Figure 1 Principle for locating the root rank where connection setup fails

After connection setup fails, each rank starts to listen to the server that can respond to fault detection connection of all ranks.
A fault detection connection request is sent to the remote end that cannot respond to the service connection setup request.
If the remote end cannot respond to the detection connection request, the link to the remote end or the service process of the remote end is faulty, and a detection failure event is generated. The event is broadcast to other connections that have been successfully set up on the server.
If a detection connection to the remote end is set up, the detection failure event sent by the remote end is received and forwarded.

In this way, if a connection fails to be set up due to any single-point fault, you can quickly locate the faulty rank based on logs and proceed to locate the fault. For details about the locating process, see Connection Setup Timeout (EI0006).

If no event is detected, the possible cause is that the behavior is inconsistent. That is, each rank has entered the connection setup phase and responded to the fault detection request of other ranks. However, the connection setup times out because communication operators called by the ranks are inconsistent. Generally, this is caused by cluster behavior consistency problems. Check factors including the script, environment, version, and dataset. To refer to the communication operator behavior, you can infer the operator behavior based on the tag information through the keyword "Alloc transports failed" in the connection setup failure logs. For example, traverse the tag information of each rank. If the communication operator is AllGather for 15 ranks but AllReduce for one rank in a 16-rank communicator, focus on analysis of the call logic differences between the two operators.

In the connection setup timeout scenario, you can quickly check whether connection setup times out for all ranks. If not, you can focus on the check of ranks that do not report the connection setup timeout error. The reference command is as follows:

for i in *;do cd $i;pwd;grep -rnc "connection fail" | grep -v ":0" | wc -l; cd ..;done

Consistency Verification Mechanism

After HCCL successfully sets up a socket connection with the remote end, it exchanges information such as the operator input parameters and CANN version between the local and remote ends and verifies the information on the local end. If the verification result shows that the information is inconsistent, an error is reported in CANN logs and printed logs, with an error code returned. For details about the fault locating process, see Parameter Consistency Verification (EI0005).

In single-operator mode, to ensure performance, HCCL triggers connection setup only when a new type or algorithm operator in each communicator is called for the first time. The consistency verification is performed only after the connection is successfully set up. Therefore, this feature cannot intercept all dispatch inconsistency problems.

Error Phase Analysis

HCCL has the following common error reporting scenarios during connection setup on the parameter plane of the communication operator:

On the Atlas A3 training product/Atlas A3 inference product and Atlas A2 training product/Atlas A2 inference product, port binding fails for the device NIC. You can run the following command to check whether port binding fails. For details, see Port Binding Failure on the Parameter Plane (EI0019).
```
grep -rE "socket type\[(0|1)\].*Please check the port status and whether the port is being used by other process"
```

Socket connection setup times out on the parameter plane. You can run the following command to check whether the connection setup on the parameter plane fails. For details, see Connection Setup Timeout (EI0006).
```
grep -r "wait socket establish timeout"
```

The communication operator consistency check fails. You can run the following command to check whether the consistency check fails. For details, see Parameter Consistency Verification (EI0005).
```
grep -r "CMD information .* check fail"
```

Parent topic: Parameter Plane Connection Setup Phase