Description

HCCL provides multiple fault detection functions, including the link setup fault detection time configuration, cluster heartbeat monitoring switch, and process suspension detection switch. After these detection functions are enabled, fault information can be quickly located and displayed when service exceptions occur, facilitating troubleshooting and handling.

The following options are supported:

connection_fault_detection_time: link setup fault detection time.
When the link setup times out, HCCL starts locating the root node where the link setup fails and propagates the information about the root node. The entire process takes the time specified by the connection_fault_detection_time parameter plus 10s required for propagating the root node information.

The value of connection_fault_detection_time can be 0 or in the range of [20, 7200]. The unit is second and the default value is 20.

If this parameter is set to 0, the link setup fault detection function is disabled. That is, when a connection fails to be set up, there is no extra waiting time and the link setup process exits immediately.
cluster_heartbeat: cluster heartbeat monitoring function, which is used to propagate fault information and record the fault root node information in runtime logs if the execution of a communication operation times out.
This parameter can be set to on (indicating to enable the heartbeat monitoring function) or off (indicating to disable the heartbeat monitoring function). The default value is on.

Note: After the cluster heartbeat monitoring function is disabled, communication operation timeouts cannot be detected, the cluster fault propagation capability will be lost, and root node fault information will not be recorded in runtime logs.
stuck_detection: process suspension detection function.
The value can be on (indicating to enable the process suspension detection function) or off (indicating to disable the process suspension detection function). The default value is on.

In scenarios that are sensitive to communication performance, you can use this parameter to disable the process suspension detection function. However, after the process suspension detection function is disabled, service suspension faults are not proactively detected and reported.
inconsistent_check: operator delivery inconsistency detection function.
The value can be on (indicating to enable the operator delivery inconsistency detection function) or off (indicating to disable the operator delivery inconsistency detection function). The default value is off.

You can use this parameter to enable the operator delivery inconsistency detection function, but the performance will deteriorate to some extent. Note that by default, after this function is disabled, the system does not proactively detect and record operator delivery inconsistency issues.

Note: This function does not support the HcclBatchSendRecv operator and graph mode scenarios. After this function is enabled, data cache is generated, occupying the host memory.
task_monitor_interval: Enables monitoring of task execution latency for communication operators under the AI CPU operator expansion mode.
The value range is [0, 7200000], in milliseconds, with a default value of 0.
- 0: Disable the task execution latency monitoring capability.
- Values greater than 0: Enables the task execution latency monitoring capability. If the execution latency of a single task exceeds the configured value, relevant task information will be printed to the directory $HOME/ascend/log/run/device-*/, marked with the log keyword StreamTaskMonitor. The timer resets after each print, so multiple rounds of logs will be output if a single task's execution latency is several times the configured threshold.
Notes:
1. This feature is only available for the Atlas A3 training product/Atlas A3 inference product and takes effect only when the communication operator expansion mode is set to AI_CPU.
2. This is an exception maintenance feature. Enabling it impacts the service execution performance, so it is not recommended for normal service running.
3. If the configured value is less than 100 milliseconds, full functionality cannot be guaranteed. This may severely degrade the service performance and impair normal functions, and even trigger service execution failures.
4. Small configured values carry a risk of log flooding under the directory $HOME/ascend/log/run/device-*/.
5. This feature serves as supplementary maintenance support for task exceptions. For such scenarios, set a value slightly lower than half the value of the HCCL_EXEC_TIMEOUT environment variable.

The detection capability provided by this environment variable is only for cluster fault localization. The detected events may not correspond to the root causes of service failures in complex scenarios. Confirm the faulty root node by combining the generation time of detected events and specific error information on the monitored node.

Example

export HCCL_DFS_CONFIG="connection_fault_detection_time:30,cluster_heartbeat:on,stuck_detection:on,inconsistent_check:off,task_monitor_interval:0"

Constraints

None

Applicability

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product (For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800T A2 training server, Atlas 900 A2 PoD cluster basic unit, and Atlas 200T A2 Box16 heterogeneous subrack are supported.)