Communicator Initialization Failure in Typical Multi-Server Scenarios (EI0015)

Symptom

The error shows up in the error log as a timeout during communicator creation when using 24 devices in a three-server cluster. Server 0 acts as the root rank. Check the error details on each server.

• Server 0: Server 0 is the root rank. A timeout occurs when the server thread waits for all ranks in the communicator to connect. From the error information, you can find the connected ranks and reversely infer that ranks 16 to 23 are unconnected.
• Server 1: This server establishes a socket connection with the root rank. After the waiting timeout occurs on the root rank, this server receives information about unconnected ranks broadcast by the root rank. The error log shows that ranks 16 to 23 are unconnected.
• Server 2: The error log on server 2 shows a timeout in the socket connection with the server. This occurs because the host network configurations on server 2 and the root rank are incorrect. Fixing these settings resolves the issue.

Fault Locating

In this scenario, if the connection times out while creating the communicator in the cluster, both the server and connected ranks can quickly find the unconnected ranks (ranks with the root cause) using the error log. You only need to identify why these ranks failed to connect. A common reason for connection timeouts is using an unconnected host NIC, which happens when the HCCL_SOCKET_IFNAME environment variable is not set.