Fault Locating

When a communication operator is called, HCCL creates a socket connection through the parameter plane network based on TCP to exchange information such as addresses based on service requirements. If some ranks do not call the expected communication operator due to a fault, the connection request cannot be initiated. Or, due to network connectivity and behavior consistency problems, some ranks cannot respond to connection requests with each other. In this case, socket connection timeout errors will be reported on other ranks.

Due to the HCCL algorithm and operator call sequence, connection timeouts often propagate through ranks in a cascading manner. Therefore, if a connection timeout occurs, you need to locate the fault point first.

HCCL provides the following mechanism in the parameter plane connection setup phase to assist in quickly locating faults.

Mechanism for Positioning Connection Setup Root Rank

Connection setup problems are propagated through ranks in a cascading manner. Assume that the connection setup between rank 0 and rank 1 times out and the connection setup between rank 1 and rank 2 times out. If the connection setup between rank 1 and rank 2 fails due to network or other reasons, a connection setup timeout error is also reported on rank 0 to indicate that connection setup between rank 0 and rank 1 fails. However, the root cause of the failure is that connection setup between rank 1 and rank 2 fails. Therefore, it is difficult to locate the root rank where the connection setup fails in the cluster. HCCL immediately starts fault detection connection after connection setup fails on the service. The implementation principles are as follows:

Figure 1 Principle for locating the root rank where connection setup fails

After connection setup fails, each rank starts to listen to the server that can respond to fault detection connection of all ranks.
A fault detection connection request is sent to the remote end that cannot respond to the service connection setup request.
If the remote end cannot respond to the detection connection request, the link to the remote end or the service process of the remote end is faulty, and a detection failure event is generated. The event is broadcast to other connections that have been successfully set up on the server.
If a detection connection to the remote end is set up, the detection failure event sent by the remote end is received and forwarded.

In this way, if a connection fails to be set up due to any single-point fault, you can quickly locate the faulty rank based on logs and proceed to locate the fault. For details about the locating process, see Connection Setup Timeout (EI0006).

If no event is detected, the possible cause is that the behavior is inconsistent. That is, each rank has entered the connection setup phase and responded to the fault detection request of other ranks. However, the connection setup times out because communication operators called by the ranks are inconsistent. Generally, this is caused by cluster behavior consistency problems. Check factors including the script, environment, version, and dataset. To refer to the communication operator behavior, you can infer the operator behavior based on the tag information through the keyword "Alloc transports failed" in the connection setup failure logs. For example, traverse the tag information of each rank. If the communication operator is AllGather for 15 ranks but AllReduce for one rank in a 16-rank communicator, focus on analysis of the call logic differences between the two operators.

In the connection setup timeout scenario, you can quickly check whether connection setup times out for all ranks. If not, you can focus on the check of ranks that do not report the connection setup timeout error. The reference command is as follows:

for i in *;do cd $i;pwd;grep -rnc "connection fail" | grep -v ":0" | wc -l; cd ..;done

Consistency Verification Mechanism

After HCCL successfully sets up a socket connection with the remote end, it exchanges information such as the operator input parameters and CANN version between the local and remote ends and verifies the information on the local end. If the verification result shows that the information is inconsistent, an error is reported in the CANN logs and screen logs and an error code is returned. For details about the fault locating process, see Parameter Consistency Verification (EI0005).

In single-operator mode, to ensure performance, HCCL triggers connection setup only when a new type or algorithm operator in each communicator is called for the first time. The consistency verification is performed only after the connection is successfully set up. Therefore, this feature cannot intercept all delivery inconsistency problems.

Error Phase Analysis

HCCL has the following common error reporting scenarios during connection setup on the parameter plane of the communication operator:

Port binding fails for the device NIC. You can run the following command to check whether port binding fails. For details, see Port Binding Failure on the Parameter Plane (EJ0003).
```
grep -rE "socket type\[(0|1)\].*Please check the port status and whether the port is being used by other process"
```

Socket connection setup times out on the parameter plane. You can run the following command to check whether the connection setup on the parameter plane fails. For details, see Connection Setup Timeout (EI0006).
```
grep -r "wait socket establish timeout"
```

The communication operator consistency check fails. You can run the following command to check whether the consistency check fails. For details, see Parameter Consistency Verification (EI0005).
```
grep -r "CMD information .* check fail"
```

Parent topic: Parameter Plane Connection Setup Phase