Timeout Occurs When a Socket Is Created Between a Rank and a Server (EI0015)

Symptom

The CANN logs contain the keyword "topo exchange agent get socket timeout," as shown below:

[ERROR] HCCL(7988,all_reduce_test):2025-03-19-04:16:13.978.979 [topoinfo_exchange_agent.cc:190] [7988][InitGroupStage][RanktableDetect]topo exchange agent get socket timeout! timeout[120] 
[ERROR] HCCL(7988,all_reduce_test):2025-03-19-04:16:13.978.995 [topoinfo_exchange_agent.cc:41] [7988][TopoInfoExchangeAgent][Setup]TopoExchangeAgent: connect server[127.10.0.1 : 60000] failed

Possible Cause

During communicator creation through cluster negotiation, the current rank creates a socket based on rootInfo as well as the IP address and port number of the server. However, the socket creation times out due to network connectivity problems.

Solution

1. Check the connectivity of the host network and the corresponding port.

   The host usually has multiple NICs. By default, HCCL selects a NIC in lexicographic order for socket connection. This may result in selecting an unconnected NIC. Use the HCCL_SOCKET_IFNAME environment variable to select a specific NIC. If the right NIC is selected, check whether the specified port is connected. For instance, check whether port 60000 on 127.10.0.1 connects to the rank with the reported error, as shown in the following log. Run the following command to query the host NIC information obtained by the current process:

   grep -r "get host ip success\|find nic.*success"

2. Check whether the interval for calling the communicator creation API of each agent and server exceeds the timeout period.

   Run the grep -r "Entry-" command in the run directory of CANN logs to check the call time, or calculate the call interval of each agent and server based on the error log printing time. If the interval exceeds the timeout period, set the connection establishment timeout period using HCCL_CONNECT_TIMEOUT. The default value is 120 seconds.

   Run the following command to query the call time of each communicator creation API:

   grep -r "Entry-HcclGetRootInfo\|Entry-HcclCommInitRootInfoInner" run/plog

   According to the following query result, the call time of the communicator creation API of rank[3] is about 200 seconds slower than that of other ranks. However, the connection setup timeout period is 120 seconds. As a result, the process of creating a communicator times out. If the service processes on ranks are different, for example, some processes need to load more data and therefore start more slowly, you can increase the timeout period by setting the HCCL_CONNECT_TIMEOUT environment variable.

   [INFO] HCCL(3079955,all_reduce_test):2025-11-20-11:59:56.716.583 [op_base.cc:1293] [3079955]Entry-HcclCommInitRootInfoInner:ranks[4], rank[3], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[3]
   [INFO] HCCL(3079952,all_reduce_test):2025-11-20-11:56:36.704.523 [op_base.cc:858] [3079952]Entry-HcclGetRootInfo:rootInfo[0xaaaae85c79a0], deviceLogicId[0]
   [INFO] HCCL(3079952,all_reduce_test):2025-11-20-11:56:36.711.546 [op_base.cc:1293] [3079952]Entry-HcclCommInitRootInfoInner:ranks[4], rank[0], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[9127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[0]
   [INFO] HCCL(3079953,all_reduce_test):2025-11-20-11:56:36.712.024 [op_base.cc:1293] [3079953]Entry-HcclCommInitRootInfoInner:ranks[4], rank[1], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[1]
   [INFO] HCCL(3079954,all_reduce_test):2025-11-20-11:56:36.712.065 [op_base.cc:1293] [3079954]Entry-HcclCommInitRootInfoInner:ranks[4], rank[2], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[2]