Timeout Occurs When a Socket Is Created Between a Rank and a Server (EI0015)
Symptom
The CANN logs contain the keyword "topo exchange agent get socket timeout!", as shown in the following:
[ERROR] HCCL(7988,all_reduce_test):2025-03-19-04:16:13.978.979 [topoinfo_exchange_agent.cc:190] [7988][InitGroupStage][RanktableDetect]topo exchange agent get socket timeout! timeout[120] [ERROR] HCCL(7988,all_reduce_test):2025-03-19-04:16:13.978.995 [topoinfo_exchange_agent.cc:41] [7988][TopoInfoExchangeAgent][Setup]TopoExchangeAgent: connect server[127.10.0.1 : 60000] failed
Possible Cause
During communicator creation through cluster negotiation, the current rank creates a socket based on rootInfo and the IP address and port number of the server. However, the socket creation times out due to network connectivity problems.
Solution
- Check the connectivity of the host network and the corresponding port.
Generally, the host has multiple NICs. By default, HCCL selects a host NIC in lexicographic order for socket connection. Therefore, a unconnected host NIC may be selected. You can use the HCCL_SOCKET_IFNAME environment variable to specify the NIC to be selected. If the NIC is correctly selected, check whether the specified port is connected. For example, check whether port 60000 of 127.10.0.1 can be connected to the rank where the error is reported based on the following log. You can run the following command to query the host NIC information obtained by the current process:
grep -r "get host ip success\|find nic.*success"
- Check whether the interval for delivering the communicator creation API of each agent and server exceeds the timeout interval.
You can run the grep -r "Entry-" command in the run directory of CANN logs to check the delivery time of the communicator creation API, or calculate the interval for delivering the communicator creation API of each agent and server based on the printing time of the error log. If the interval exceeds the timeout interval, you can set the connection establishment timeout interval using HCCL_CONNECT_TIMEOUT. The default value is 120 seconds.
You can run the following command to query the delivery time of each communicator creation API:
grep -r "Entry-HcclGetRootInfo\|Entry-HcclCommInitRootInfoInner" run/plog
According to the following query result, the delivery time of the communicator creation API of rank[3] is about 200 seconds slower than that of other ranks. However, the connection setup timeout interval is 120 seconds. As a result, the entire process of creating a communicator times out. In this scenario, if the service processes on different ranks are different, for example, some processes need to load more data and therefore start more slowly, you can increase the timeout interval by setting the HCCL_CONNECT_TIMEOUT environment variable to solve the problem.
[INFO] HCCL(3079955,all_reduce_test):2025-11-20-11:59:56.716.583 [op_base.cc:1293] [3079955]Entry-HcclCommInitRootInfoInner:ranks[4], rank[3], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[3] [INFO] HCCL(3079952,all_reduce_test):2025-11-20-11:56:36.704.523 [op_base.cc:858] [3079952]Entry-HcclGetRootInfo:rootInfo[0xaaaae85c79a0], deviceLogicId[0] [INFO] HCCL(3079952,all_reduce_test):2025-11-20-11:56:36.711.546 [op_base.cc:1293] [3079952]Entry-HcclCommInitRootInfoInner:ranks[4], rank[0], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[9127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[0] [INFO] HCCL(3079953,all_reduce_test):2025-11-20-11:56:36.712.024 [op_base.cc:1293] [3079953]Entry-HcclCommInitRootInfoInner:ranks[4], rank[1], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[1] [INFO] HCCL(3079954,all_reduce_test):2025-11-20-11:56:36.712.065 [op_base.cc:1293] [3079954]Entry-HcclCommInitRootInfoInner:ranks[4], rank[2], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[127.10.0.1%enp_60000_0_1763610996711234], deviceLogicId[2]