文档
下载

Connection Setup Timeout (EI0006)

The HCCL connection setup timeout is affected by the HCCL_CONNECT_TIMEOUT environment variable. If the remote end cannot respond to the service connection setup request within the timeout period, "socket timeout" is reported. In addition, if the remote end exits due to faults such as timeout, "recv fail" may also be reported during data exchange on the established connection.

Symptom

The CANN logs contain the keyword "wait socket establish timeout" or "[InitChannelStage][Timeout]," as shown below:

[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.403 [hccl_socket_manager.cc:797] [18744][Wait][LinkEstablish]wait socket establish timeout, role[1] rank[1] timeout[120 s]
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.454 [hccl_socket_manager.cc:861] [18744][Wait][LinksEstablishCompleted] is failed. ret[9].
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.646 [hccl_socket_manager.cc:623] [18744]   _________________________LINK_ERROR_INFO___________________________
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.650 [hccl_socket_manager.cc:624] [18744]   |  comm error, device[1]
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.653 [hccl_socket_manager.cc:626] [18744]   |  dest_ip(user_rank)  |   dest_port   |  src_ip(user_rank)   |   src_port   |   MyRole   |   Status   |    TlsStatus   |
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.655 [hccl_socket_manager.cc:628] [18744]   |----------------------|---------------|----------------------|--------------|------------|------------|----------------|
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.706 [hccl_socket_manager.cc:583] [18744]   |  192.0.2.199(0)   |  16666  |   192.0.3.198(1)   |  3234403008  |  client  | time out |   DISABLE  | LinkInfo
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.942 [hccl_socket_manager.cc:836] [18744][Create][Sockets]Wait links establish completed failed, local role is client. ret[9][ERROR] HCCL(17528,python3):2026-03-18-10:33:52.113.964 [transport_manager.cc:1402] [18744][SetMachinePara]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:33:52.114.027 [transport_manager.cc:1252] [18744][CreateLink][InitChannelStage][Timeout]SetMachinePara error.
[ERROR] HCCL(17528,python3):2026-03-18-10:34:34.224.286 [detect_connect_anomalies.cc:494] [20039][CreateClientConnect]GetStatus fail, ret[9]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.141.949 [detect_connect_anomalies.cc:127] [18744]-------------------CONNECT TIMEOUT DETECT RESULT-----------------------
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.141.966 [detect_connect_anomalies.cc:132] [18744]This node (server 192.168.200.100, device ID 1) detects that srcRank (server 192.168.200.100, device ID 1) fails to connect to dstRank (server 192.168.200.100, device ID 0). Continue to analyze the fault based on the logs of srcRank and dstRank.
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.141.970 [detect_connect_anomalies.cc:135] [18744]1. If the link setup timeout is reported on both ends, check the network connectivity between the two ends.2. If dstRank reports other exceptions, locate the cause based on the exception information of dstRank.3. If dstRank does not report any error, the possible cause is that the service process is suspended or exits in advance
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.141.977 [detect_connect_anomalies.cc:143] [18744]----------------------------------------------------------------------
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.013 [transport_manager.cc:1325] [18744][InitChannelStage][Timeout]Transport init error! createLink para:rank[1]-localUserrank[1]-localIpAddr[192.168.200.100/1], remoteRank[0]-remoteUserrank[0]-remoteIpAddr[192.168.200.100/0], machineType[1], linkMode[1], isUsedRdma[0], tag[HcomAllReduce_6629421139219749105_0]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.040 [transport_manager.cc:1214] [18744][TransportManager][PrintErrorInfo]local rank information: nicType[VNIC_TYPE], logicSuperPodId is not set, phySuperPodId[287454020].
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.095 [transport_manager.cc:256] [18111][checkSubCommLinkThreadsStatus]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.101 [transport_manager.cc:363] [18111][AllocSubCommLinks]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.105 [transport_manager.cc:672] [18111][Alloc]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.108 [hccl_communicator_host.cc:6370] [18111][AllocAlgResource]Alloc transports failed, tag[HcomAllReduce_6629421139219749105_0_device]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.120 [hccl_communicator_host.cc:4325] [18111][HcclCommunicator][ExecOp] AllocAlgResource failed, algName=[AllReduceRingFor91093Executor]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.145 [hccl_communicator_host.cc:2858] [18111][AllReduce]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.152 [hccl_comm.cc:306] [18111][HcclComm][HcomAllReduce_6629421139219749105_0]errNo[0x0000000000000009] index[0]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.156 [hcom.cc:515] [18111][AllReduce][Result]errNo[0x0000000005010009] hcclComm AllReduce error, tag[HcomAllReduce_6629421139219749105_0], input_ptr[0x12e083e00200], output_ptr[0x12e086600400], count[10485888], data_type[float32], op[sum]
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.164 [hcom_ops_kernel_info_store.cc:807] [18111][HcomAllReduceOpKernel]call trace: hcclRet -> 9
[ERROR] HCCL(17528,python3):2026-03-18-10:34:43.142.169 [hcom_ops_kernel_info_store.cc:358] [18111][HCCLOpsKernel]call trace: hcclRet -> 9

Determining the Remote End to Check Based on the Logs

  • If "DETECT EVENT LIST" is recorded in the error logs, pay attention to the failed connection pair in the logs. In the preceding log example, check the root cause of the connection setup failure between device 7 and device 6 of 127.10.0.1 displayed in the exception event DETECT EVENT[1].
  • If "DETECT EVENT LIST" is not recorded, obtain the device IP addresses of the two ends from the LINK_ERROR_INFO table in the error logs. Then, obtain the information about the remote and local ranks from the key log information Transport init error! createLink para:. The format is [hostIp/deviceId].

    Run the grep -r "Transport init error! createLink para:" debug/plog/plog-*.log command. The following information is displayed:

    [ERROR] HCCL(3215542,all_reduce_test):2025-11-20-18:18:03.114.306 [transport_manager.cc:886] [3215599][InitChannelStage][Timeout]Transport init error! createLink para:rank[2]-localUserrank[2]-localIpAddr[127.10.0.1/2], remoteRank[1]-remoteUserrank[1]-remoteIpAddr[127.10.0.1/1], machineType[1], linkMode[1], isUsedRdma[0], tag[AllReduce_127.10.0.1%enp_60000_0_1763633852475745
    • localUserrank: local rank ID
    • localIpAddr: IP address of the local rank
    • remoteUserrank: remote rank ID
    • remoteIpAddr: IP address of the remote rank
    • tag: communication operator identifier

After obtaining the information about the remote end that fails to establish a connection, further analyze the problem based on the CANN logs of the two ends.

Checking Whether Behavior Inconsistency Exists Across Devices on the Remote End

Connection setup on the parameter plane is an interactive process between two ends. The connection can be successfully set up only when both ends initiate connection setup requests within the timeout period. Otherwise, an error is reported due to waiting timeout. Therefore, you can find the rank information of the remote end based on the error information reported by the local end and view the logs of the remote end for further judgment.

Figure 1 Troubleshooting process

Check point 1:

If no error log is generated on the remote end, the remote end may not dispatch the corresponding communication operator synchronously. As a result, the local end cannot receive feedback for the connection setup request from the remote end, causing a waiting timeout.

Check whether the communication operator dispatch behavior is consistent on the two ends.

Check point 2:

If other errors occur on the remote end except the parameter plane connection setup timeout, locate the cause of the errors on the remote end.

Check point 3:

If the remote end reports a parameter plane connection setup timeout but the error information indicates that the remote end is setting up a connection with another rank instead of the local end, locate the cause of the parameter plane connection setup timeout on the remote end first.

Check point 4:

If a timeout occurs when the local end and remote end set up a connection on the parameter plane, check whether the error reporting time of the two ends exceeds the connection setup timeout period. If so, check the root cause of the communication operator dispatch timeout on the two ends.

The connection setup timeout period can be specified by HCCL_CONNECT_TIMEOUT. The default value is 120 seconds. You can run the grep -r "HCCL_CONNECT_TIMEOUT" run/plog/ command in the run directory of CANN logs to query the timeout period configured for the current service.

Check point 5:

If the parameter plane connection setup timeout period of the remote end and local end is within the connection setup timeout period, check the network connectivity of the two ends.

  1. Check whether the TLS configurations of the two ends are consistent. If not, the verification fails during socket creation. As a result, the connection setup at both ends times out. You can use the following methods to check the TLS configurations of the two ends:
    • In the LINK_ERROR_INFO table of the error log, status shows the TLS status of the current device. UNKNOWN means that the status is not obtained, DISABLE means that TLS is disabled, and ENABLE means that TLS is enabled.
    • Run the grep -r "TLS SWITCH" log/run/device-* command in the log of the rank to obtain the TLS status.
      run/device-0/device-2849330_20251024153927364.log:[INFO] HCCP(2988,hccp_service.bin):2025-10-24-15:39:26.133.826 [rs_ssl.c:1529]tid:2988,rs_ssl_init(1529) : TLS SWITCH (1)
run/device-1/device-2849331_20251024153928174.log:[INFO] HCCP(30877,hccp_service.bin):2025-10-24-15:39:25.142.466 [rs_ssl.c:1529]tid:30877,rs_ssl_init(1529) : TLS SWITCH (0)
    • Run for i in {0..7}; do hccn_tool -i $i -tls -g ; done | grep switch to check the TLS configuration of the rank using hccn_tool.
      # for i in {0..1}; do hccn_tool -i $i -tls -g ; done | grep switch
dev_id:0, tls switch[0](0:disable, 1:enable), tls preconfigured[1](0:non-preset, 1:preset), tls alarm time threshold[60]days
dev_id:1, tls switch[1](0:disable, 1:enable), tls preconfigured[1](0:non-preset, 1:preset), tls alarm time threshold[60]days
  2. If the two ends for connection setup are on different ranks, check the network connectivity between the device network ports of the local and remote ends by running the hccn_tool command on one rank to ping the device IP address of the other rank.
    hccn_tool -i {node} -ping -g address {Remote end IP address}

    If the two ranks cannot be pinged or a network port is down, contact the lab administrator to check the configuration of the corresponding NIC and switch.

  3. If SuperPoDs of the Atlas A3 training product/Atlas A3 inference product are used, check whether ranks on the physical SuperPoDs are incorrectly configured as a logical SuperPoD. If so, HCCL incorrectly considers that the two nodes can communicate with each other through the vNICs on the SuperPoD. As a result, the mutual waiting times out.

    Check the link type and physical SuperPoD information of both ends based on the following logs. The link type is vNIC, the physical SuperPoD IDs of both ends are different (0 and 1), but the logical SuperPoD IDs are the same (logic_1). The timeout occurs when the vNIC link is selected for communication. You can modify or cancel the HCCL_LOGIC_SUPERPOD_ID configuration to resolve this issue.

    Local log

    debug/plog/plog-3003627_20260205184335411.log:14:[ERROR] HCCL(3003627,scatter_test):2026-02-05-18:44:26.379.547 [transport_manager.cc:885] [3003959][TransportManager][PrintErrorInfo]local rank information: nicType[VNIC_TYPE], logicSuperPodId[logic_1], phySuperPodId[0]. Note: Do not configure ranks belonging to different physical superpod ID info a single logical superpod ID

    Remote log:

    debug/plog/plog-3003628_20260205184354321.log:14:[ERROR] HCCL(3003628,scatter_test):2026-02-05-18:44:26.379.542 [transport_manager.cc:885] [3003959][TransportManager][PrintErrorInfo]local rank information: nicType[VNIC_TYPE], logicSuperPodId[logic_1], phySuperPodId[1]. Note: Do not configure ranks belonging to different physical superpod ID info a single logical superpod ID

Note:

  1. The default threshold for generating a detection failure event for the faulty connection is 20s. You can adjust the threshold by setting the connection_fault_detection_time field in the HCCL_DFS_CONFIG environment variable. The value 0 disables this function. If you have a large cluster or serious device synchronization issues, increase this field value for accurate detection results.
  2. In some complex service scenarios, both connection setup and execution timeouts may occur in a service. You need to perform multiple redirections based on the detection results to locate the fault. Check the rank logs to determine whether the root rank has been reached. Generally, the root rank has other errors or has no exception log, or a timeout occurs when the root rank waits for other ranks.
我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情