Documentation
Download

Required Knowledge and Skills

Before locating a fault, ensure that you are familiar with the basic concepts of HCCL and the fault locating assistance function.

HCCL error codes cover most common problems. If the error information does not contain an error code or has the error code EI9999, the fault may be rare or an HCCL internal problem may occur. Analyze the fault based on the CANN logs and code. If you cannot rectify the fault, contact technical support.

For a fault without a clear first error in a large cluster, you need to sort out the behaviors of each rank and find the root rank based on the dependency between ranks. To solve this problem, HCCL provides the capabilities of locating the connection root rank and monitoring cluster heartbeats, and provides diagnosis results in FAQs. For details about the principles, see Fault Locating and Cluster Heartbeat Mechanism.

This document applies to the following scenarios:

  • The description of the HCCL implementation mechanism in this document is used only to explain the mechanisms of various faults and assist in analyzing the fault symptom and locating the fault cause. If the content of the running mechanism is inconsistent with that in the related document, refer to the document of the running mechanism first.
  • Some CANN log examples in this document will be adjusted with version updates. You can focus on the key information in the logs. If there is a big difference, use the log information.
  • When an HCCL exception occurs in the service, the error log information of the HCCL component is recorded in the CANN logs. If no error log of the HCCL component is found in the CANN logs, check whether error information of other components exists. If no error is reported, check whether the training script is abnormal, whether core dump occurs, or whether the process is suspended.

Environment Variables Related to Fault Diagnosis

  • HCCL_CONNECT_TIMEOUT and HCCL_EXEC_TIMEOUT

    The environment variables indicate the timeout periods of the HCCL in the connection setup phase and execution phase. It is recommended that the value of HCCL_CONNECT_TIMEOUT be less than that of HCCL_EXEC_TIMEOUT to ensure that the first error information can be correctly reported in complex scenarios, thus determining whether the abnormal service process is blocked at the local or remote end.

  • HCCL_ENTRY_LOG_ENABLE

    This environment variable specifies whether to enable recording of HCCL operator-level input parameters. If the cause of a cluster behavior consistency problem cannot be located by other means, you can enable this environment variable to record the collective communication behavior on different ranks and find the behavior difference introduction point through horizontal comparison between devices.

  • HCCL_DEBUG_CONFIG

    This environment variable specifies whether to enable HCCL module-level logging. During operator development and debugging, you can use this configuration to analyze algorithm selection, task orchestration, and other log information inside the operator.

  • HCCL_DFS_CONFIG

    This environment variable configures HCCL advanced fault detection. For details, see the environment variable description. The default value is recommended.

HCCL Log Description

HCCL log information is recorded in CANN logs. For details about CANN logs, see Log Reference.

  • When an HCCL error is reported, key fault information is recorded in the debug directory of CANN logs. If some training frameworks are used, HCCL also records key error information in service logs.
  • By default, HCCL records some key run logs in the run directory of CANN logs, such as communicator initialization and destruction (printed by default) and communication operator dispatch (requiring enabling of the HCCL_ENTRY_LOG_ENABLE environment variable). The following are examples of key logs:
    • Communicator initialization
      Entry-HcclGetRootInfo:rootInfo[0x7fffcd65f130], deviceLogicId[0]
Entry-HcclCommInitRootInfoConfigInner:ranks[16], rank[0], rootinfo: host ip[127.10.0.1] port[60000] nicDeploy[1] identifier[group_name_0], deviceLogicId[0]
      • ranks: size of the communicator.
      • rank: ID of the current rank in the communicator.
      • rootinfo: root rank information.
      • identifier: communicator name.
    • Communicator destruction
      Entry-HcclCommDestroy: op_base comm destroy begin
    • Communication operator dispatch (requires enabling the HCCL_ENTRY_LOG_ENABLE environment variable)
      Entry-HcclAllReduce: tag[AllReduce_127.10.0.1%eth1_30000_0_1736576907435382], sendBuf[0x12e7bf550000], recvBuf[0x12e7bf550000], count[531260224], dataType[float32], op[sum], localRank[0], streamId[5],comm[0x331c9c00], deviceLogicId[0]
      • tag: communication operator identifier
      • sendBuf: input data address pointer.
      • recvBuf: output data address pointer.
      • count: data size.
      • dataType: data type.
      • op: reduction calculation type.
      • localRank: local rank ID.
      • streamId: communication operator execution stream.
      • comm: communicator pointer.
      • deviceLogicid: logical device ID dispatched by the communication operator.
    • To quickly search for and identify the information about the communicator and the local end, HCCL provides the quick search keywords Communicator Key Info and LocalRank Key Info.
      • For example, run the grep -r "Communicator Key Info" command to obtain the following information:
        run/plog/plog-858941_20251210195327204.log:[INFO] HCCL(858941,all_reduce_test):2025-12-10-19:53:28.131.350 [hccl_communicator_attrs.cc:327] [858941][Communicator Key Info]identifier[127.0.0.1%enp_60000_0_1765367607599032] rankSize[8] serverNum[1] moduleNum[1] superPodNum[0] multiModuleDiffDeviceNumMode[0] multiSuperPodDiffServerNumMode[0]

        Key information communicator information: identifier (communicator name), rankSize (communicator size), serverNum (number of ranks in the communicator), moduleNum (number of modules in the communicator), superPodNum (number of SuperPoDs in the communicator), multiModuleDiffDeviceNumMode (whether the number of devices is inconsistent across modules), and multiSuperPodDiffServerNumMode (whether the number of nodes is inconsistent across SuperPoDs). In the information, 1 indicates yes and 0 indicates no.

      • For example, run the grep -r "LocalRank Key Info" command to obtain the following information:
        run/plog/plog-858941_20251210195327204.log:[INFO] HCCL(858941,all_reduce_test):2025-12-10-19:53:28.131.357 [hccl_communicator_attrs.cc:330] [858941][LocalRank Key Info]userRank[6] hostIp[127.0.0.1] devicePhyId[6] server[127.0.0.1] deviceIp[0.0.0.0] superPodId[0] useSuperPodMode[0] isStandardCard[0]

        Key local end information: userRank (rank ID in the communicator), hostIp (IP address on the host), devicePhyId (physical ID), server (rank information), deviceIp (IP address on the device), superPodId (SuperPoD ID), useSuperPodMode (whether the SuperPoD mode is used), and isStandardCard (whether the standard card scenario is used). In the information, 1 indicates yes and 0 indicates no.

    • Configurations and effective values of environment variables are printed in the run/plog directory of the CANN logs.
      For the Atlas inference product, Atlas training product, Atlas A2 training product/Atlas A2 inference product and Atlas A3 training product/Atlas A3 inference product, you can search for the keyword HCCL_ENV to query the effective values of environment variables on each process. For example, run the grep -r "HCCL_ENV" run/plog/plog-xxx.log command to obtain the following information:
      [INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.877 [externalinput.cc:598] [1595259][HCCL_ENV] HCCL_CONNECT_TIMEOUT set by default to [120]s
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.882 [externalinput.cc:558] [1595259][HCCL_ENV] HCCL_EXEC_TIMEOUT set by default to [1836]s
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.886 [externalinput.cc:663] [1595259][HCCL_ENV] HCCL_INTRA_PCIE_ENABLE set by default to [1], HCCL_INTRA_ROCE_ENABLE set by default to [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.890 [externalinput.cc:742] [1595259][HCCL_ENV] environmental variable PROFILING_MODE and GE profiling option is not set, default: false
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.895 [externalinput.cc:833] [1595259][HCCL_ENV] HCCL_WHITELIST_DISABLE set by environment to [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.912 [externalinput.cc:880] [1595259][HCCL_ENV] HCCL_IF_IP is not set
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.915 [externalinput.cc:936] [1595259][HCCL_ENV] HCCL_SOCKET_IFNAME set by default to [EmptyString]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.917 [externalinput.cc:903] [1595259][HCCL_ENV] HCCL_SOCKET_FAMILY is not set and is used by default [AF_INET]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.920 [externalinput.cc:865] [1595259][HCCL_ENV] HCCL_IF_BASE_PORT set by default to [60000]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.988 [externalinput.cc:1170] [1595259][HCCL_ENV] HCCL_RDMA_TC set by default to [132]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.881.991 [externalinput.cc:1205] [1595259][HCCL_ENV] HCCL_RDMA_SL set by default to [4]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.058 [externalinput.cc:1250] [1595259][HCCL_ENV] HCCL_RDMA_TIMEOUT set by default to [20]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.064 [externalinput.cc:1284] [1595259][HCCL_ENV] HCCL_RDMA_RETRY_CNT set by default to [7]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.069 [externalinput.cc:1370] [1595259][HCCL_ENV] HCCL_BUFFSIZE set by environment to [1]M
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.072 [externalinput.cc:621] [1595259][HCCL_ENV] HCCL_DETERMINISTIC set by default to [false]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.074 [externalinput.cc:1395] [1595259][HCCL_ENV] HCCL_DIAGNOSE_ENABLE set by default to [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.077 [externalinput.cc:1484] [1595259][HCCL_ENV] HCCL_ENTRY_LOG_ENABLE set by default to [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.081 [externalinput.cc:1505] [1595259][HCCL_ENV] HCCL_INTER_HCCS_DISABLE is not set, default value is FALSE.
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.090 [externalinput.cc:1569] [1595259][HCCL_ENV] environmental variable HCCL_OP_EXPANSION_MODE is [HOST], aicpuUnfold[0], aivMode[0], enableFfts[1]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.096 [externalinput.cc:1420] [1595259][HCCL_ENV] HCCL_RDMA_QPS_PER_CONNECTION is set to default value [1]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.099 [externalinput.cc:1454] [1595259][HCCL_ENV] HCCL_MULTI_QP_THRESHOLD is set to default value [512]KB
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.116 [externalinput.cc:1724] [1595259][HCCL_ENV][ParseRetryEnable] HCCL_OP_RETRY_ENABLE set by environment variable to [L0:0,L1:0,L2:0].
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.120 [externalinput.cc:1736] [1595259][HCCL_ENV] HCCL_OP_RETRY_PARAMS is not set, default value MaxCnt is [1], HoldTime is [5000]ms, IntervalTime is [1000]ms
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.123 [externalinput.cc:1778] [1595259][HCCL_ENV] HCCL_LOGIC_SUPERPOD_ID set by environment to [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.125 [externalinput.cc:525] [1595259][HCCL_ENV] HCCL_RDMA_PCIE_DIRECT_POST_NOSTRICT set by default to [EmptyString], rdmaFastPost is [0]
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.128 [externalinput.cc:791] [1595259][HCCL_ENV][Parse][MultiQpSrcPortConfigPath]environmental variable HCCL_RDMA_QP_PORT_CONFIG_PATH is empty
[INFO] HCCL(1595259,alltoall_test):2026-01-06-15:38:29.882.131 [externalinput.cc:1800] [1595259][HCCL_ENV] HCCL_DEBUG_CONFIG is not set, debugConfig set by default to 0x0
      For the Atlas 350 Accelerator Card, you can search for the keyword base_config to query the configured environment variables.
      [INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.170[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_IF_IP" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.176[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_IF_BASE_PORT" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.181[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_SOCKET_IFNAME" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.187[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_WHITELIST_DISABLE" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.192[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_HOST_SOCKET_PORT_RANGE" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.197[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_SOCKET_FAMILY" is not set. Default value is used. 
[INFO] HCCL(229424,python3.8):2025-12-23-22:31:40.239.206[base_config.cc:33][229424][Init][EnvVarParam]Env config "HCCL_CONNECT_TIMEOUT" is parsed.
Parent topic: Fault Locating
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details