Fault Locating

After HCCL communication operator tasks are orchestrated, they are dispatched to the device for asynchronous execution. If a task fails to be executed, the device calls the callback function to notify HCCL of the abnormal task information (stream and task ID). HCCL searches for the task dispatch information and prints details of the failed task and its operator. To trace task information on the Atlas A3 training product/Atlas A3 inference product and Atlas A2 training product/Atlas A2 inference product, enable HCCL_DIAGNOSE_ENABLE.

[ERROR] HCCL(2111667,all_reduce_test):2025-10-24-11:18:29.597.044 [task_exception_handler.cc:908] [2111667][TaskExecStage][Timeout][Host]Task run failed, base information is streamID:[2], taskID[21], tag[AllReduce_127.10.0.1%enp_60000_0_1761275812718970], AlgType(level 0-1-2):[fullmesh-ring-NHR].
[ERROR] HCCL(2111667,all_reduce_test):2025-10-24-11:18:29.597.054 [task_exception_handler.cc:771] [2111667][TaskExecStage][Timeout][Host]Task run failed, groupRank information is group:[127.10.0.1%enp_60000_0_1761275812718970], user define information[], rankSize[4], rankId[2].
[ERROR] HCCL(2111667,all_reduce_test):2025-10-24-11:18:29.597.083 [task_exception_handler.cc:704] [2111667][TaskExecStage][Timeout][Host]Task run failed, opData information is timeStamp:[2025-10-24-11:16:55.490.253], deviceId[2], index[21], count[256], reduceType[sum], src[0x12c0c0013000], dst[0x12c0c0014000], dataType[float32].

First, identify the key information of the communication operator from the task exception.

• base information: stream, task ID, and tag of the HCCL operator. You can identify the HCCL operator that reports the error based on the tag.
• groupRank information: communicator name (group), communicator size (rankSize), and rank ID of the current device in the communicator.
• opData information: input parameter, device ID, index in the communicator, data size (count), reduction type (reduceType), and input (src) and output (dst) addresses of the current operator.

Generally, only two types of tasks may fail:

• Notify: This type of task may fail when the waiting for the remote end times out during operator execution.
• SDMA: This type of task may fail in scenarios such as HCCS connection exception and multi-bit ECC. There is a low probability that this type of task fails when a core dump occurs on the remote end.