Quick Fault Locating and Demarcation

• In common error scenarios, HCCL prints error and fault information in service logs. If the service logs contain the error code EI**** or EJ****, rectify the fault based on the fault information, or based on the error information in the CANN logs. For details about the error codes, see HCCL Error Codes.
• Except for error codes, HCCL prints error logs of the HCCL component in the CANN logs. If no error log of the HCCL component is found in the CANN logs, check whether error information of other components exists. If no error is reported, check whether the training script is abnormal, whether core dump occurs, or whether the process is suspended.

HCCL collective communication is a global collaborative behavior in a communicator. If an HCCL error is reported on a rank, the possible cause is that a timeout occurs when waiting for a remote end. Locate the root cause based on the log information of the remote end. To locate and demarcate HCCL problems, collect CANN logs of all ranks in the cluster, including logs in the debug and run directories.

The HCCL service has three phases: communicator initialization, parameter plane connection setup, and communication operator execution. The hardware resources, communication topology, and synchronization mode used in different phases are significantly different. Therefore, you need to determine the phase where the HCCL error is reported and then locate the fault based on the phase.

• HCCL provides multi-level search keywords for common error scenarios. You can quickly identify the error reporting phase based on the keywords in the error logs and further locate the fault based on the error information. For details about multi-level search keywords, see HCCL Multi-level Search Keywords. The following log indicates that a timeout error is reported during operator execution and the current operator is expanded in host mode:

  [ERROR] HCCL(858209,all_reduce_test):2025-12-10-19:52:32.589.097 [task_exception_handler.cc:27] [858274][TaskExecStage][Timeout][HOST]Task run failed, base information is streamID:[1740], taskID[23], tag[AllReduce_127.0.0.1%enp_60000_0_1765367469951573], AlgType(level 0-1-2):[fullmesh-ring-NHR].

  Note: The multi-level search keyword function is supported only in CANN 8.5.0 and later versions. For versions that do not support this function or scenarios where no keyword is found, you can determine the error reporting phase using other methods.

• HCCL provides the communicator creation API and communication operator API, which are dispatched synchronously and executed asynchronously. The following scenarios are involved:
  • If the service fails to call the communicator creation API or the error log contains the topoinfo or ranktable keyword, rectify the fault by referring to Communicator Initialization Phase.
  • If the service fails to call the communication operator API or the error log contains the transport keyword, rectify the fault by referring to Parameter Plane Connection Setup Phase.
  • If the communicator creation API and communication operator API are both successfully dispatched, but the HCCL operator fails to be executed when stream synchronization is triggered or the error log contains the TaskExceptionHandler, FFTS+ run failed, or Task run failed keyword, rectify the fault by referring to Task Dispatch and Execution Phase.

    In addition to the key information in the three phases, if the service log contains a specific error code, for example, EI0001, you can find the corresponding error code in the subsequent content and further locate the fault.