Some Ranks Are Not Connected to the Server (EI0015)

Symptom

The CANN logs contain the keyword "topo exchange server get socket timeout!" or "Failed to connect agent."

• Symptom on the server:

  [ERROR] HCCL(1041081,all_reduce_test):2025-11-21-01:20:01.624.966 [topoinfo_exchange_server.cc:314] [1041362][InitGroupStage][RanktableDetect]topo exchange server get socket timeout! timeout[120 s]
  [ERROR] HCCL(1041081,all_reduce_test):2025-11-21-01:20:01.625.103 [topoinfo_exchange_server.cc:501] [1041362][InitGroupStage][DisplayConnectionedRank]total connected num is [4],line num is [1]
  [ERROR] HCCL(1041081,all_reduce_test):2025-11-21-01:20:01.625.112 [topoinfo_exchange_server.cc:503] [1041362][InitGroupStage][DisplayConnectionedRank]need connect rankNum is [8]
  [ERROR] HCCL(1041081,all_reduce_test):2025-11-21-01:20:01.625.145 [topoinfo_exchange_server.cc:517] [1041362][InitGroupStage][DisplayConnectionedRank]connected rankinfo[LINE 0]: [0000000000000000],[0000000000000002],[0000000000000004],[0000000000000006];

• Symptom on the agent:

  [ERROR] HCCL(1041085,all_reduce_test):2025-11-21-01:20:01.630.122 [topoinfo_exchange_base.cc:145] [1041085][InitGroupStage][RanktableDetect] TopoDetect ERROR occur !!! fault_type[1], fault_info["Failed to connect agent[1,3,5,7,]"]
  [ERROR] HCCL(1041085,all_reduce_test):2025-11-21-01:20:01.630.557 [topoinfo_exchange_agent.cc:552] [1041085][InitGroupStage][RanktableDetect]rank num[8]is different with rank list size[4] in total topo rank info.

Possible Cause

After the server calls the HcclGetRootinfo API, a background thread is started to wait for all ranks to connect until the timeout period expires. If some ranks in the communicator fail to connect to the server thread within the timeout period, the server thread reports a timeout error and prints the connected rank list. You can locate the ranks that fail to be connected based on this list and further locate the connection failure cause.

Solution

Figure 1 Fault locating when some ranks are not connected

1. Locate the ranks that are not connected based on the error information.
   • Server: After a waiting timeout occurs, the server prints information about the connected ranks. The rank IDs in the communicator are in the sequence of [0, rankSize – 1]. Therefore, you can calculate the ranks that are not connected based on the information about the connected ranks. The preceding log example indicates that rank 9 is not connected. You need to further locate the cause why rank 9 is not connected.
   • Agent: Connected agents will receive the error information broadcast by the server to indicate unconnected ranks in the cluster after the waiting timeout occurs on the server. You can directly confirm the unconnected ranks based on "Failed to connect agent" in the error information and further confirm the failure cause. The preceding log example indicates that rank 9 is not connected. You need to further locate the cause why rank 9 is not connected.
2. Check whether the communicator creation API is called for unconnected ranks.

   During communicator creation, HCCL records default logs in the CANN run directory. You can query the logs in the entire cluster to check whether the communicator of a rank is dispatched. For example, run the following command to check whether the communicator of rank 9 is created and dispatched. If there are multiple communicators, there may be several log records. Check whether the communicators are the same based on the communicator name identifier in the logs.

   grep -r "Entry-HcclCommInitRootInfoInner" | grep "rank\[9\]"

   • If the communicator creation API is called for a rank, obtain the node and process ID of the unconnected rank based on the search result, and then check for the error information in the CANN logs of the unconnected rank.
   • If the cluster logs do not contain information indicating that the communicator creation API is called for a rank, check the service to determine the reason why the communicator creation API is not called for the rank.