Service Process and Locating Method

Service Process

During cluster information negotiation (in the scenario where a communicator is created based on the root rank information), HCCL sets up a socket connection with each rank through the server to exchange local information. This helps obtain the cluster information of the entire communicator and initialize the communicator.

Figure 1 Service Process

The server calls the HcclGetRootInfo API to start the listening thread.
1. Select a server in the cluster, which is usually rank 0 in the communicator. The rank calls the HcclGetRootInfo API.
2. Obtain the IP address and port number of the host NIC to generate the rootInfo value. The host NIC can be specified by the HCCL_SOCKET_IFNAME environment variable, and the port number can be specified by the HCCL_IF_BASE_PORT and HCCL_HOST_SOCKET_PORT_RANGE environment variables.
3. The server performs binding and listening based on the IP address and port number, starts a background thread to wait for all agents in the communicator to be connected, and directly returns rootInfo to complete the API call.
The rank calls the HcclCommInitRootInfo API to set up a connection with the server.
1. The upper-layer service or framework layer broadcasts rootInfo to each rank in the communicator. Each rank calls the HcclCommInitRootInfo API and uses rootInfo as the input parameter.
2. A rank uses the host NIC to set up a socket connection with the server and sends its own rankInfo to the server. After the sending is complete, the rank enters the RX state and waits for the server to return the complete cluster information. In this phase, the socket connection setup and cluster information waiting must be completed within a specified timeout period, which can be controlled by the HCCL_CONNECT_TIMEOUT environment variable.
The server collects full cluster information and sends the information to each rank.
1. After collecting rankInfo of all ranks, the background thread of the server generates complete cluster information and sends the information to each agent thread of the rank. In this way, each rank obtains all rank information in the entire communicator.
2. In addition, the server waits for the socket connections of all ranks to be completed within a certain timeout period. The timeout period can be controlled by the HCCL_CONNECT_TIMEOUT environment variable.

Possible Cause

In the mode of creating a communicator based on the root rank information, socket connections need to be set up between ranks. The NIC on the host is used, and the operation needs to be synchronously performed for all ranks in the communicator within the timeout period. Therefore, the common cause of the negotiation failure is that the network connection of the host NIC is abnormal or the socket connection is not correctly set up for some ranks. The first step for fault locating is to find the abnormal rank.

Common causes and key logs of the cluster information negotiation failure are as follows:

Port binding fails for the server. You can run the following command to check whether port binding fails. For details, see Port Binding Failure on the Server (EI0019).
```
grep -r "socket type\[2\].*Please check the port status and whether the port is being used by other process"
```
The server does not receive socket connections from all ranks in the communicator. You can run the following command to quickly query the information. For details, see Some Ranks Are Not Connected to the Server (EI0015).
```
grep -r "topo exchange server get socket timeout"
```
A timeout occurs when setting up a socket connection between the rank and the server. You can run the following command to quickly query the information. For details, see Timeout Occurs When a Socket Is Created Between a Rank and a Server (EI0015).
```
grep -r "topo exchange agent get socket timeout"
```

Key Log Information

During communicator creation, HCCL records key log information in the plog file in the run directory of the CANN logs when calling the communicator creation API. You can check whether a process calls the corresponding communicator creation API based on the corresponding logs. For details about the logs, see HCCL Log Description.
If some ranks fail to set up socket connections with the root rank in the communicator negotiation phase, the root rank prints information about connected ranks before exiting due to timeout and broadcasts the missing rank information to the connected ranks. You can locate unconnected ranks based on the information to further locate the root cause. For details about the logs, see Some Ranks Are Not Connected to the Server (EI0015).

Parent topic: Cluster Information Negotiation