(Optional) Configuring Fault Detection Levels

Configuration Description

For different fault codes of hardware faults, chip faults, UnifiedBus device faults, and common faults, resumable training provides default fault levels and corresponding fault handling policies. For chip faults, it also provides default fault frequency and duration settings, along with the corresponding handling policies.

If you need to modify the fault handling policy, refer to this section. Do not modify it unless there are special requirements.

Supported Configurable Fault Levels

The fault levels that can be configured for different types of faults are shown in the table below.

Table 1 Configurable fault levels

Fault Name	Configurable Fault Levels
Node fault	NotHandleFault, PreSeparateFault, SeparateFault
Chip fault	NotHandleFault, RestartRequest, RestartBusiness, FreeRestartNPU, RestartNPU, SeparateNPU, PreSeparateNPU, SubHealthFault
UnifiedBus device fault	NotHandleFault, SubHealthFault, ResetFault, SeparateFault , RestartRequestFault
Common fault	NotHandleFault, SeparateNPU, SubHealthFault , PreSeparateNPU

In the table above, the handling policy for each fault level is described as follows.

Table 2 Fault levels and handling policies

Fault Handling Policy	Description	Rescheduling	Graceful Fault Tolerance
NotHandleFault	Has no service impact and requires no handling.	Not handled for now	Not handled for now
RestartRequest	Affects service execution and requires re-executing the service request.	Isolate the chip and reschedule the job. Note: If the inference job subscribes to fault information, and a RestartRequest fault occurs on the inference card used by the job with a fault duration not exceeding 60 seconds, job rescheduling will not be performed. If the fault duration exceeds 60 seconds without recovery, the chip will be isolated and job rescheduling will be performed.	In inference scenarios, re-execute the inference request; in training scenarios, re-execute the training service.
RestartBusiness	Affects service execution and requires re-executing the service.		Re-execute the service.
FreeRestartNPU	Affects service execution and requires resetting the chip when it becomes idle.		Wait for the chip to become idle and then reset it.
RestartNPU	Affects service execution and requires resetting the chip immediately.		Immediately stop the training service, reset the chip, and then re-execute the service.
SeparateNPU	Unrecoverable and requires isolating the chip.		Isolate the chip and reschedule the job.
SeparateFault	The job will definitely be affected. Note: When the UnifiedBus device fault level is SeparateFault, it indicates that the service has failed to run, and the component or board needs to be replaced.	Reschedule the job. [!NOTE] Note For UnifiedBus device faults, the fault handling policy represented by this fault level is to stop the current training job, isolate the node, and reschedule the job.	-
RestartRequestFault	The service has failed to run and requires re-executing the service request.	Stop the current training job, isolate the node, and reschedule the job.	In inference scenarios, re-execute the inference request; in training scenarios, re-execute the training service.
ResetFault	The service has failed to run.	Stop the current training job, isolate the node, and reschedule the job.	-
PreSeparateNPU	Does not affect the service for now, but jobs will no longer be scheduled to this chip subsequently.	Pre-isolate the chip.	Pre-isolate the chip.
PreSeparateFault	May cause the job to be affected.	If there is a job on this node, it will not be handled. During subsequent scheduling, jobs will not be scheduled to this node.	-
SubHealthFault	Handled based on the value of the subHealthyStrategy parameter configured in the job YAML. For details, see YAML Configuration Description.	When a sub-health fault occurs on the chip, it needs to be handled according to the Configuring YAML. Note: If a fault of another level occurs on the chip subsequently, the SubHealthFault handling policy will not affect the handling of faults at other levels.	Handled according to the policy.

Node Hardware Faults

Configuration File Description

Resumable training performs hierarchical processing for different levels of node hardware faults. NodeD obtains the fault code of the current fault and processes the fault accordingly based on the fault level configured for the fault code in NodeDConfiguration.json. The supported fault levels and handling methods for node hardware faults are described as follows.

The NodeD configuration file NodeDConfiguration.json is a system configuration file. Do not modify it arbitrarily unless you have special requirements. If you need to modify the fault level of a fault code, you can do so through the mindx-dl-node-fault-config file created from NodeDConfiguration.json. For operation instructions, see (Optional) Configuring Node Hardware Fault Levels.For fault level descriptions and node status descriptions, see Customizing Node Faults.

(Optional) Configuring Node Hardware Fault Levels

When creating a NodeD image, the fault level configuration file NodeDConfiguration.json is built into the image. When NodeD starts, it reads the default configuration from this file as the basis for current fault handling.

If you want to customize fault levels, create a ConfigMap file (mindx-dl-node-fault-config) in the cluster.

If mindx-dl-node-fault-config exists in the cluster when NodeD starts, NodeD will prioritize the content configured in the existing mindx-dl-node-fault-config as the basis for current fault handling.
If mindx-dl-node-fault-config exists in the cluster after reinstalling NodeD, NodeD's default NodeDConfiguration.json will not take effect, and the existing mindx-dl-node-fault-config in the cluster will be used. If you want to use the default configuration of NodeDConfiguration.json, delete mindx-dl-node-fault-config so that NodeD reads the default NodeDConfiguration.json file.
If there are issues such as format errors in the content of mindx-dl-node-fault-config, NodeD will read the content of the NodeDConfiguration.json file built into the image by default as the basis for current fault handling.

Procedure

Taking fault code 0100001D as an example, the following shows how to modify the handling policy for the current fault from NotHandleFault (no handling required) to PreSeparateFault (do not handle if there are jobs on the node, and do not schedule subsequent jobs to the node).

Run the following command to create the ConfigMap file (mindx-dl-node-fault-config) required for dynamic fault level configuration.

kubectl create cm mindx-dl-node-fault-config -n mindx-dl  --from-file=./NodeDConfiguration.json

Command output:

configmap/mindx-dl-node-fault-config created

Table 1 Parameter description

Parameter Name	Description
mindx-dl-node-fault-config	Name of the created ConfigMap file. This file name cannot be modified.
mindx-dl	Namespace name. This namespace cannot be modified.
NodeDConfiguration.json	Used to configure fault codes and their corresponding fault levels. Must be consistent with the NodeDConfiguration.json file name.

Run the following command to edit the mindx-dl-node-fault-config file.
```
kubectl edit cm -n mindx-dl mindx-dl-node-fault-config
```
In the mindx-dl-node-fault-config file, locate the fault code 0100001D.
```
 "FaultTypeCode": {
        "NotHandleFaultCodes":[
          "0100001D","03000009","03000013","0300000D","03000011"
        ],
...
  ],
...
```
During fault level customization, if the following issues occur accidentally, this modification will be invalid, and NodeD will use the last saved configuration for processing.
- The file format is abnormal or the fault code is incorrect. The fault code can only be an 8-character string containing digits and letters.
- The same fault code is configured in multiple fault levels at the same time.

Delete the fault code 0100001D from NotHandleFaultCodes and add it to PreSeparateFaultCodes.

 "FaultTypeCode": {
        "NotHandleFaultCodes":[
         "03000009","03000013","0300000D","03000011"
        ],
        "PreSeparateFaultCodes":[
          "28000037","00000011", "0100001D"
...
        ],
...

After the modification is complete, press Esc, enter :wq! to save and exit.
After the mindx-dl-node-fault-config file is updated, check whether the operation is successful.
1. Run the following command to query the log name of NodeD.
```
kubectl get pods -A | grep noded
```
  Command output:
```
mindx-dl      noded-c5f52   1/1     Running   0               2m16s
```
2. Query the log information of NodeD by using the queried log name.
```
kubectl logs noded-c5f52 -n mindx-dl -f
```
  If the log contains "update fault config success", it indicates that the dynamic fault code configuration operation is successful.

Chip Faults

Overview

Both Ascend Device Plugin and ClusterD provide the capability to manually isolate chips based on fault frequency. The functional differences between the two are as follows:

Ascend Device Plugin determines faults based on the node dimension and counts the frequency of actual faults that occur.
ClusterD determines faults based on the job dimension.
- If multiple chips under a single job experience the same fault simultaneously within 30 seconds, it excludes a hardware fault, and does not count the fault frequency. This judgment rule applies to most scenarios. For scenarios such as a Pod being deleted but residual processes remaining, the fault frequency counts may have deviations.
- Only new faults can trigger the judgment of whether the fault frequency for manual chip isolation has reached the upper limit. If the configured threshold is adjusted to the current count, isolation will not be triggered immediately; the judgment logic will only be triggered when the next fault occurs.
- After ClusterD restarts, the frequency count information will be lost, and the fault frequency for manual chip isolation will start counting from zero.
- If job scheduling does not meet expectations after removing the isolation, check whether the node has the label huawei.com/scheduler.chip1softsharedev.enable=false. If this label exists, delete it.

The fault codes involved in the manual chip isolation feature of Ascend Device Plugin and ClusterD theoretically do not need to be duplicated. If you do not want to use the isolation feature of Ascend Device Plugin, see the (Optional) Configuring Chip Fault Frequency and Duration section to delete the manual chip isolation-related configuration in the faultCustomization.json file. If you do not want to use the isolation feature of ClusterD, see the (Optional) Configuring Chip Fault Frequency section to disable manual chip isolation.

If both Ascend Device Plugin and ClusterD have manually isolated the same chip, the isolation must be removed separately for each. For the method to remove isolation in Ascend Device Plugin, see "Manually Recovering Force-Isolated Chips" in (Optional) Configuring Chip Fault Frequency and Duration. For the method to remove isolation in ClusterD, see "Manually Recovering Manually Isolated Chips" in (Optional) Configuring Chip Fault Frequency.

Ascend Device Plugin

Configuration File Description

For chip faults, resumable training supports processing based on fault level, fault frequency, and fault duration configuration.

When performing hierarchical processing for different levels of chip faults, Ascend Device Plugin obtains the fault code of the current fault and processes the fault accordingly based on the fault level configured for the fault code in faultCode.json.
When processing based on the fault frequency and duration of chip faults, Ascend Device Plugin obtains the fault code of the current fault and processes the fault accordingly based on the fault frequency and duration configured for the fault in faultCustomization.json.

faultCode.json and faultCustomization.json are system configuration files. Do not modify them arbitrarily unless you have special requirements. If the default frequency fault configuration of Ascend Device Plugin contains faults that can be triggered by software reasons, you can delete the corresponding fault code yourself. (Software reasons may cause a certain fault to repeatedly occur a large number of times within a short period under a single job, causing Ascend Device Plugin to detect that the fault has reached the fault frequency and place a large number of devices into manual isolation state.)

If you need to modify the fault level corresponding to a fault code, you can do so through the mindx-dl-fault-config file created from faultCode.json and faultCustomization.json.

For the fault code corresponding to each fault, see the Chip Fault Code References section.
For the fault levels that can be configured for chip faults, see Fault Levels.
For the fault frequency and duration that can be configured for chip faults, see Fault Frequency and Duration.

Fault Levels in faultCode.json

Resumable training performs hierarchical handling for different levels of chip faults. If you need to modify the fault level of a fault code, see (Optional) Configuring Chip Fault Levels for operation instructions.

After Ascend Device Plugin obtains the chip fault code from the driver, it classifies the fault into several levels based on the impact of the fault code on the device and service. For details, see Table 1.

The training process must be stopped before chip reset; otherwise, the reset will fail.
If Ascend Device Plugin receives an unrecognized fault code (not saved in faultCode.json) through subscription, it performs fault handling according to the handling suggestion provided by the subscription interface by default. If the fault level received by the subscription interface is Hint or Minor, it is handled at the NotHandleFault level; if the fault level is any other level, it is handled at the SeparateNPU level.

Fault Frequency and Duration

Resumable training handles the fault frequency and duration of chip faults. Certain hardware faults may occur repeatedly during a single training job, causing the training job to be interrupted and rescheduled repeatedly. The cluster scheduling components provide an initialization configuration file, faultCustomization.json, to elevate the fault level for the fault codes corresponding to these faults.

The relationship between the initialization configuration in the faultCustomization.json file and fault types is described in Initialization Configuration and Fault Types.
For the default configuration (default values) of the faultCustomization.json file, see Table 2.
If you need to modify the fault frequency and duration configuration, see (Optional) Configuring Chip Fault Frequency and Duration for operation instructions.

Initialization Configuration and Fault Types

The current faultCustomization.json file only provides initialization configuration for upgrading the fault level of identifiable hardware faults.

If the following fault occurs three times within 24 hours, the chip fault level is upgraded to ManuallySeparateNPU, a fault level that requires manual intervention. For details, see faultCustomization.json Parameter Description.

The following example uses the fault name HBMC Ca Parity Error, corresponding to fault code 80E18005, to escalate the current fault level to ManuallySeparateNPU (a fault level that requires manual intervention).

  "FaultFrequency": [
    {
      "EventId": [
        "80C98000","80B78000","80B58000","80A18008","80A38008","80A58008","80B98000","80B98008","80BB8000",
        "80BB8008","80BD8000","80BD8008","80C78008","80C98008","80CB8008","80CD8008","80CF8008","80D98008",
        "80DF8008","80DE1801","80E01801","80E18008","80E38008","80E39200","80E3A202","80E3A203","80E78000",
        "80E78008","80F18000","80F18008","80F38008","80F78008","81318008","81338008","813B8008","81478008",
        "81578008","815F8008","81938008","81958008","81978008"
      ],
      "TimeWindow": 86400,
      "Times": 2,
      "FaultHandling": "ManuallySeparateNPU"
    },
    {
      "EventId": ["80E18005"],
      "TimeWindow": 86400,
      "Times": 3,
      "FaultHandling": "ManuallySeparateNPU"
    }
  ],

When the fault handling policy is ManuallySeparateNPU, you can refer to the steps in "Manually Recovering Force-Isolated Chips" in (Optional) Configuring Chip Fault Frequency and Duration for processing.
In addition to identifiable hardware faults, the faultCustomization.json file also includes the following types of faults.
- Faults that do not require handling: These faults do not affect training jobs or devices, and no initialization configuration for upgrading the fault level is provided.
- Faults that cannot be identified as hardware or software faults: These faults cannot be accurately identified as hardware or software faults and will affect training jobs. No initialization configuration for upgrading the fault level is provided for such faults. You are advised to manually configure the maximum number of resumable training times supported and the fault handling policy after the maximum number is reached based on the actual situation. For details, see (Optional) Configuring Chip Fault Frequency and Duration.
- Software configuration faults: These faults are software configuration issues and do not occur under normal circumstances. No initialization configuration for upgrading the fault level is provided for such faults. You are advised to check whether the software versions are compatible.

faultCustomization.json Parameter Description

When there is no need to manually modify the faultCustomization.json file, Ascend Device Plugin performs fault handling according to the default configuration (default values) of faultCustomization.json. For the parameter description of the faultCustomization.json file, see Table 2.

(Optional) Configuring Chip Fault Levels

When the Ascend Device Plugin image is built, the faultCode.json and faultCustomization.json configuration files are built into the image. When Ascend Device Plugin starts, it reads the default configurations of these two files as the basis for current fault handling. For the description of faultCode.json and faultCustomization.json, see Configuration File Description.

If you want to customize the fault level or graceful fault tolerance related configurations, create a ConfigMap file (mindx-dl-fault-config) in the cluster.

If mindx-dl-fault-config exists in the cluster when Ascend Device Plugin starts, Ascend Device Plugin will prioritize the content configured in the existing mindx-dl-fault-config as the basis for current fault handling.
If mindx-dl-fault-config exists in the cluster after Ascend Device Plugin is reinstalled, the default faultCode.json of Ascend Device Plugin will not take effect, and the existing mindx-dl-fault-config in the cluster will be used instead.
If you want to use the default configuration of faultCode.json or faultCustomization.json, you can delete mindx-dl-fault-config so that Ascend Device Plugin reads the default faultCode.json, SwitchFaultCode.json, or faultCustomization.json files.
If there are issues such as format errors in the ConfigMap file content, Ascend Device Plugin will read the content of the built-in ConfigMap file in the image by default as the basis for current fault handling.

Configuring Fault Levels Using faultCode.json

Take the fault dmp_daemon (node status detection anomaly), corresponding to fault code 80E21007, as an example. The following shows the operation to modify the current fault handling policy from NotHandleFaultCodes (no handling required) to RestartNPUCodes (isolate the chip and perform job rescheduling).

Run the following command to create the ConfigMap file (mindx-dl-fault-config) required for dynamically configuring fault codes.

kubectl create cm mindx-dl-fault-config -n kube-system --from-literal="PollInterval=300" --from-file=./faultCode.json

Command output:

configmap/mindx-dl-fault-config created

Table 1 Parameter description

Parameter Name	Required	Description
mindx-dl-fault-config	Yes	The name of the ConfigMap file used for dynamically configuring fault codes. This file name cannot be modified.
kube-system	Yes	The namespace where mindx-dl-fault-config resides. This namespace name cannot be modified.
PollInterval	No	If this parameter is not specified, the default value is 300s. It specifies the polling interval for checking whether the mindx-dl-fault-config file has been updated. The unit is seconds, and the value range is 30 to 3600. Changes to PollInterval will take effect in the next polling cycle.
faultCode.json	Yes	Used to store fault codes. It must be consistent with the faultCode.json file name.

Run the following command to edit the mindx-dl-fault-config file.
```
kubectl edit cm -n kube-system mindx-dl-fault-config
```
In the mindx-dl-fault-config file, locate the fault code 80E21007.
```
"NotHandleFaultCodes":[

"80E21007","80E38003","80F78006","80C98006","80CB8006","81318006","80A18006","80A18005","80FB8000","8C1F8609",
...
  ],
...
```
If the same fault code is configured in multiple fault levels, the configuration will be displayed as successful, but the fault will be handled according to the higher-level fault by default.

Delete the fault code 80E21007 from NotHandleFaultCodes and add it to RestartNPUCodes.

"NotHandleFaultCodes":[
     "80E38003","80F78006","80C98006","80CB8006","81318006","80A18006","80A18005","80FB8000","8C1F8609",
...
  ],
...
"RestartNPUCodes":[
   "8C204E00","A8028802","A4302003","A4302004","A4302005","A4302006","A4302009","A430200A","80CF8009","80CF8008","80E21007",...
...
   ],

After modification, press Esc, type :wq! to save and exit.
Wait for the mindx-dl-fault-config file update to take effect (PollInterval defaulted to 300s if not specified), then check whether the operation is successful.
1. Run the following command to query the log name of Ascend Device Plugin.
```
kubectl get pods -A | grep ascend-device-plugin
```
  Command output:
```
kube-system      ascend-device-plugin-daemonset-910-jmlf5   1/1     Running   0              6h34m
```
2. Use the queried log name to query the component log information of Ascend Device Plugin.
```
kubectl logs -n kube-system ascend-device-plugin-daemonset-910-jmlf5
```
  If the log shows "load fault code from configmap success", it indicates that the manual fault code configuration operation was successful.

(Optional) Configuring Chip Fault Frequency and Duration

When the Ascend Device Plugin image is built, the faultCode.json and faultCustomization.json configuration files are built into the image. When Ascend Device Plugin starts, it reads the default configurations from these two files as the basis for current fault handling. For descriptions of faultCode.json and faultCustomization.json, see Configuration File Description.

If you want to customize the chip fault frequency and duration, create a ConfigMap file (mindx-dl-fault-config) in the cluster.

If mindx-dl-fault-config exists in the cluster when Ascend Device Plugin starts, Ascend Device Plugin will prioritize the content configured in the existing mindx-dl-fault-config as the basis for current fault handling.
If mindx-dl-fault-config exists in the cluster after reinstalling Ascend Device Plugin, the default faultCustomization.json of Ascend Device Plugin will not take effect, and the existing mindx-dl-fault-config in the cluster will be used. If you want to use the default configuration of faultCustomization.json, you can delete mindx-dl-fault-config so that Ascend Device Plugin reads the default faultCustomization.json file.
If there are issues such as format errors in the ConfigMap file content, Ascend Device Plugin will read the content of the built-in ConfigMap file in the image as the basis for current fault handling by default.

Modifying the fault frequency is a high-risk operation. Improper modification may cause chips to be mistakenly isolated. For example, software faults caused by job errors may occur repeatedly in large numbers within a short period, causing Ascend Device Plugin to detect that the fault frequency has been reached, and then place a large number of chips into the manual isolation state, making a large number of nodes unschedulable.

Procedure

Taking fault code 80CB8002 as an example, if this fault occurs repeatedly on a certain chip, causing the training service to be rescheduled repeatedly, you can manually configure the supported maximum number of resumable training times within 24 hours to 2. The fault handling policy is ManuallySeparateNPU after the maximum number is reached.

Log in to the environment and go to the Ascend Device Plugin decompression directory.
Run the following command to check whether mindx-dl-fault-config has been created based on the faultCode.json file.
```
kubectl describe cm -n kube-system mindx-dl-fault-config
```
- If mindx-dl-fault-config exists and contains the relevant fields of faultCustomization.json, perform Step 4 to edit the file.
- If mindx-dl-fault-config exists but does not contain the relevant fields of faultCustomization.json, save the content of mindx-dl-fault-config first, delete the mindx-dl-fault-config file, and then perform Step 3 to create the file.
- If mindx-dl-fault-config does not exist, perform Step 3 to create it.

Run the following command to create the ConfigMap file (mindx-dl-fault-config) required for chip fault frequency configuration.

kubectl create cm mindx-dl-fault-config -n kube-system --from-literal="PollInterval=300" --from-file=./faultCode.json --from-file=./faultCustomization.json

Command output:

configmap/mindx-dl-fault-config created

Table 1 Parameter description

Parameter Name	Required	Description
mindx-dl-fault-config	Yes	The ConfigMap file name required for dynamically configuring fault codes. This file name cannot be modified.
kube-system	Yes	The namespace where mindx-dl-fault-config resides. This namespace name cannot be modified.
PollInterval	No	If this parameter is not specified, the default value is 300s. It specifies the polling interval for checking whether the mindx-dl-fault-config file is updated, in seconds. The value ranges from 30 to 3600. Modifications to PollInterval take effect in the next polling cycle.
faultCode.json	Yes	Used to store fault codes. The file name must be consistent with faultCode.json.
faultCustomization.json	No	Used to customize configurations such as graceful fault tolerance time, fault frequency, and fault duration (only parameter plane network faults are supported). If this parameter is not specified, there is no fault frequency configuration, and other configurations are processed using default values. The file name must be consistent with faultCustomization.json.

Run the following command to edit the mindx-dl-fault-config file.

kubectl edit cm -n kube-system mindx-dl-fault-config

Modify the chip fault frequency and duration based on the actual situation.

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
PollInterval: "300"
# Modify the fault level of chip faults
faultCode.json: |
{
"NotHandleFaultCodes":[
...
}
# Modify the fault frequency and duration of chip faults
faultCustomization.json: |
{
 "GraceTolerance": {
 "WaitProcessReadCMTime": 30,
 "WaitDeviceResetTime": 150,
 "WaitFaultSelfHealingTime": 15
},
"FaultFrequency": [
 {
  "EventId": [
    "80C98000","80B78000","80B58000","80A18008","80A38008","80A58008","80B98000","80B98008","80BB8000",
    "80BB8008","80BD8000","80BD8008","80C78008","80C98008","80CB8008","80CD8008","80CF8008","80D98008",
    "80DF8008","80DE1801","80E01801","80E18008","80E38008","80E39200","80E3A202","80E3A203","80E78000",
    "80E78008","80F18000","80F18008","80F38008","80F78008","81318008","81338008","813B8008","81478008",
    "81578008","815F8008","81938008","81958008","81978008"
  ],
  "TimeWindow": 86400,
  "Times": 2,
  "FaultHandling": "ManuallySeparateNPU"
 },
 {
  "EventId": ["80E18005"],
  "TimeWindow": 86400,
  "Times": 3,
  "FaultHandling": "ManuallySeparateNPU"
 },
 {
  "EventId": ["81078603"],
  "TimeWindow": 86400,
  "Times": 5,
  "FaultHandling": "ManuallySeparateNPU",
  "ReleaseTimeWindow": 172800
 }
],
"FaultDuration": [
 {
  "EventId": ["81078603"],
  "FaultTimeout": 20,
  "RecoverTimeout": 60,
  "FaultHandling": "PreSeparateNPU"
 },
 {
  "EventId": ["81B18603"],
  "FaultTimeout": 5,
  "RecoverTimeout": 60,
  "FaultHandling": "PreSeparateNPU"
 }
]
}
kind: ConfigMap
metadata:
creationTimestamp: "2024-06-20T10:12:07Z"
name: mindx-dl-fault-config
namespace: kube-system
resourceVersion: "52893696"
selfLink: /api/v1/namespaces/kube-system/configmaps/mindx-dl-fault-config

In the mindx-dl-fault-config file, add the following code under the FaultFrequency field to set the maximum number of resumable training times supported for fault 80CB8002 within 24 hours to 2, and the handling policy for the fault after reaching the maximum number to ManuallySeparateNPU.
```
{
  "EventId": ["80CB8002"],
  "TimeWindow": 86400,
  "Times": 2,
  "FaultHandling": "ManuallySeparateNPU"
}
```
After modification, press Esc and enter :wq! to save and exit.
After the mindx-dl-fault-config file update takes effect (PollInterval defaulted to 300s if not specified), check whether the operation is successful.
1. Run the following command to query the log name of Ascend Device Plugin.
```
kubectl get pods -A | grep ascend-device-plugin
```
  Command output:
```
kube-system      ascend-device-plugin-daemonset-910-jmlf5   1/1     Running   0              6h34m
```
2. Use the queried log name to query the log information of Ascend Device Plugin.
```
kubectl logs -n kube-system ascend-device-plugin-daemonset-910-jmlf5
```
  - If the log contains "load fault customization from configmap complete", it indicates that the manual chip fault frequency configuration operation is successful.
  - If the log contains "modify xxx success", it indicates that the xxx parameter in faultCustomization.json in the ConfigMap is set successfully.
  - If the log contains "insert fault frequency success", it indicates that the occurrence time of a frequency fault has been recorded. Within the frequency window, after the number of fault records for that chip reaches the fault frequency trigger threshold, the corresponding fault level will be reported.
(Optional) Manually restore a forcibly isolated chip. When the fault handling policy is ManuallySeparateNPU, the chip remains isolated after fault recovery. If the release conditions are not met and you need to manually restore the forcibly isolated chip, do as follows.
1. Run the following command to find device-info-cm reported by Ascend Device Plugin of this node.
```
kubectl get cm -n kube-system | grep deviceinfo | grep {nodeName}
```
2. Run the following command to edit device-info-cm.
```
kubectl edit cm -n kube-system {configMapName}
```
3. Delete the name of the recovered, healthy chip following ManuallySeparateNPU under data.
```
apiVersion: v1
kind: ConfigMap
data:
  DeviceInfoCfg: '{"DeviceInfo":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-1,Ascend910-2,Ascend910-3,Ascend910-4,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-Fault":"[]","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Unhealthy":""},"UpdateTime":1718702470},"CheckCode":"4f00cf1d220da26a8fdbeb5ba163a751d4b264c48b81d22149257e272ae3b413"}'
  ManuallySeparateNPU: Ascend910-0
```
  Delete all chip names after the ManuallySeparateNPU field and set the value to empty "".
4. After modification, press Esc and enter :wq! to save and exit.
5. Wait for one reporting cycle (if device information changes, it will be reported within the health status check cycle; if device information does not change, the reporting cycle is fixed at 5 minutes), then run the following command to check whether the chip name just deleted exists in ManuallySeparateNPU in device-info-cm. If it does not exist, the chip has successfully recovered to a healthy state and can continue to be used normally.
```
kubectl describe cm -n kube-system {configMapName}
```

ClusterD

Configuration Description

Resumable training can handle chip faults based on the fault frequency configuration.

When performing hierarchical processing for different levels of chip faults, ClusterD obtains the fault code and fault level of the current fault. For faults at levels other than NotHandleFault and SubHealthFault, the chip status is set to manual isolation based on the fault frequency configured in the ConfigMap (clusterd-config-cm). For parameter descriptions of this ConfigMap, see Table 1.

clusterd-config-cm is a system configuration. Do not modify it arbitrarily unless you have special requirements. If you need to modify the detection switch for manually isolated chips, fault frequency, isolation removal time, etc., you modify this ConfigMap by referring to (Optional) Configuring Chip Fault Frequency Configuration.
Configuring the fault code detection range is not supported. ClusterD makes judgments based on the fault level reported by Ascend Device Plugin. For faults at levels other than NotHandleFault and SubHealthFault, they will all be included in the manual isolation chip detection process.

(Optional) Configuring Chip Fault Frequency

When ClusterD is installed, the ConfigMap (clusterd-config-cm) is automatically created as the detection basis for manually isolated chips. For parameter descriptions of this ConfigMap, see Table 1.

If you want to customize the chip fault frequency, you can modify this ConfigMap. If the modified ConfigMap content has format errors or other issues, ClusterD will retain the last successfully read configuration as the detection basis for manual chip isolation. If the ConfigMap content read by ClusterD at startup is incorrect, the manual chip isolation detection mechanism will be disabled by default until the format and content are correct.

Procedure

Take adjusting a manual chip isolation threshold from the default value of 3 occurrences to 5 occurrences within 24 hours as an example.

Log in to the environment and run the following command to query the current configuration.
```
kubectl describe cm -n cluster-system clusterd-config-cm
```
- If clusterd-config-cm exists, proceed to Step 3 for editing.
- If clusterd-config-cm does not exist, proceed to Step 2 for creation.
Under normal circumstances, clusterd-config-cm exists. If it does not exist, check whether there are errors in the ClusterD installation process.

Create clusterd-config-cm required for manual chip isolation detection.

Save the following content as the file cm.yaml:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: clusterd-config-cm
      namespace: cluster-system
    data:
      manually_separate_policy.conf: |
        enabled: true
        separate:
          fault_window_hours: 24
          fault_threshold: 3
        release:
          fault_free_hours: 48

Run the following command:

kubectl apply -f cm.yaml

The following example output indicates a successful creation.

configmap/clusterd-config-cm created

Run the following command to edit clusterd-config-cm.

kubectl edit cm -n cluster-system clusterd-config-cm

Modify the fault frequency for manual chip isolation based on the actual situation. For parameter descriptions, see Table 1.

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  manually_separate_policy.conf: |
    # Modify the detection switch for manually isolated chips
    enabled: true
    separate:
      # Modify the fault frequency of the manually isolated chip
      fault_window_hours: 24
      fault_threshold: 5   # Change from 3 to 5
    release:
      # Modify the De-isolation Time
      fault_free_hours: 48
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"manually_separate_policy.conf":"enabled: true\nseparate:\n  fault_window_hours: 24\n  fault_threshold: 3\nrelease:\n  fault_free_hours: 48\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"clusterd-config-cm","namespace":"cluster-system"}}
  creationTimestamp: "2026-02-24T11:25:19Z"
  name: clusterd-config-cm
  namespace: cluster-system
  resourceVersion: "3344125"
  selfLink: /api/v1/namespaces/cluster-system/configmaps/clusterd-config-cm
  uid: 68210bfc-f742-4765-a497-b61e9cc6b1a6

After the modification is complete, press the Esc key, enter :wq! to save and exit.
Wait for the clusterd-config-cm update to take effect (the detection cycle of ClusterD is 300s), and then check whether the operation is successful.
1. Run the following command to query the log name of ClusterD.
```
kubectl get pods -A | grep clusterd
```
  Command output:
```
mindx-dl      clusterd-559bf4bd6-z9hv4   1/1     Running   0             4m23s
```
2. Use the queried component log name to query the log information of ClusterD.
```
kubectl logs -f -n mindx-dl clusterd-559bf4bd6-z9hv4
```
  - If the log shows "load manually separate policy config success", it indicates that the operation to manually modify the fault frequency for manual chip isolation was successful.
  - If the log shows "node: xx, dev: xx, code: xx is not found in manual fault cache, add", it indicates that this fault triggers manual isolation.
  - If the log shows "node: xx, dev: xx, code: xx is found in manual fault cache, update last separate time", it indicates that a fault triggering manual chip isolation has once again reached the fault frequency for manual isolation, and LastSeparateTime in clusterd-manual-info-cm will be updated. For a description of clusterd-manual-info-cm, see clusterd-manual-info-cm.
(Optional) Manually recover a manually isolated chip. When the fault handling policy is ManuallySeparateNPU, the chip remains in an isolated state after fault recovery, and you can manually recover the manually isolated chip.
1. Run the following command to edit the ConfigMap clusterd-manual-info-cm.
```
kubectl edit cm -n cluster-system clusterd-manual-info-cm
```
2. Delete the name of the chip to be removed from manual isolation following the Total field under Data, for example, Ascend910-2.
```
Name:         clusterd-manual-info-cm
Namespace:    cluster-system
Labels:       <none>
Annotations:  <none>

Data
====
localhost.localdomain:
----
{"Total":["Ascend910-0","Ascend910-2","Ascend910-3"],"Detail":{"Ascend910-0":[{"FaultCode":"8C084E00","FaultLevel":"ManuallySeparateNPU","LastSeparateTime":1770811685650}],"Ascend910-2":[{"FaultCode":"8C084E00","FaultLevel":"ManuallySeparateNPU","LastSeparateTime":1770811685650}],"Ascend910-3":[{"FaultCode":"8C084E00","FaultLevel":"ManuallySeparateNPU","LastSeparateTime":1770811685650}]}}

Events:  <none>
```
3. After the modification is complete, press Esc and enter :wq! to save and exit.
4. After waiting for 15 seconds, run the following command to check whether Ascend910-2 still exists in the Total and Detail fields of clusterd-manual-info-cm. Also, check whether the ManuallySeparateNPU fault of this chip exists in cluster-info-device-\${m}. If it does not exist, the chip has been successfully removed from manual isolation and can continue to be used normally.
```
kubectl describe cm -n cluster-system clusterd-manual-info-cm
```
  - Only deletion of chips from the Total field is supported; manual addition is not supported. Modification of other content is not supported.
  - After manually recovering a chip from manual isolation, the fault count of the chip will be cleared. Manual isolation will be triggered again only when the frequency is reached again.
  - If you need to delete all manually isolated chips on a node, you must delete all chip names following the Total field and set the value to []. To remove all manually isolated chips at once, you can directly delete clusterd-manual-info-cm.
  - Within 15 seconds after ClusterD starts, do not modify clusterd-manual-info-cm temporarily to avoid data errors.

Parameter Plane Network Fault

Bus Device Fault

Configuration File Description

When performing hierarchical processing for different levels of bus device faults, Ascend Device Plugin obtains the fault code of the current fault and processes the fault according to the fault level configured for the fault code in SwitchFaultCode.json. SwitchFaultCode.json is a system configuration file. Do not modify it arbitrarily unless you have special requirements. If you need to modify the fault level corresponding to a fault code, you can do so through the mindx-dl-fault-config file created from faultCode.json and SwitchFaultCode.json.

Only Atlas A3 training series products have bus devices, and the fault codes for such devices can be viewed in the SwitchFaultCode.json file.

Fault Levels in SwitchFaultCode.json

Resumable training supports hierarchical processing for different levels of bus device faults. If you need to modify the fault level of a fault code, see (Optional) Configuring Bus Device Fault Levels for operation instructions.

After Ascend Device Plugin obtains the fault code from the driver, it classifies the fault into several levels based on the impact of the fault code on the device and service, and performs corresponding rescheduling processing. For details, see Table. Fault levels and handling policies.

(Optional) Configuring Bus Device Fault Levels

When building the Ascend Device Plugin image, the fault level configuration file SwitchFaultCode.json is built into the image. When Ascend Device Plugin starts, it reads the default configuration of this file as the basis for current fault handling.

If you want to customize the fault level or graceful fault tolerance related configuration, create a ConfigMap file (mindx-dl-fault-config) in the cluster.

If mindx-dl-fault-config exists in the cluster when Ascend Device Plugin starts, Ascend Device Plugin will preferentially use the content configured in the existing mindx-dl-fault-config as the basis for current fault handling.
If mindx-dl-fault-config exists in the cluster after Ascend Device Plugin is reinstalled, the default SwitchFaultCode.json of Ascend Device Plugin will not take effect, and the existing mindx-dl-fault-config in the cluster will be used.
If mindx-dl-fault-config exists in the cluster after Ascend Device Plugin is reinstalled and the SwitchFaultCode.json field exists in this ConfigMap, the default SwitchFaultCode.json of Ascend Device Plugin will not take effect, and the existing mindx-dl-fault-config in the cluster will be used.
If you want to use the default SwitchFaultCode.json configuration, you can delete mindx-dl-fault-config so that Ascend Device Plugin reads the default SwitchFaultCode.json file.
If there are issues such as format errors in the ConfigMap file content, Ascend Device Plugin will read the content of the built-in ConfigMap file in the image by default as the basis for current fault handling.

Using SwitchFaultCode.json to Configure Fault Levels

Take the bus device fault code [0x00f1ff09,155913,cpu,na] as an example. This fault code consists of four parts: alarm ID, fault ID, peer device type, and port number, as shown in Table 1 Fault code description.

Table 1 Fault code description

Parameter	Description	Value
Alarm ID	In the above example, the alarm ID is 0x00f1ff09.	Values in-band and out-of-band must be consistent.
Fault ID	In the above example, the fault ID is 155913.	Values in-band and out-of-band must be consistent.
Peer device type	The peer device type corresponding to this fault. In the above example, the peer device type is cpu.	na: This fault is a chip fault and does not involve a peer device. cpu: The peer device corresponding to this fault is a CPU. npu: The peer device corresponding to this fault is an NPU. L2: The peer device corresponding to this fault is an L2.
Port number	In the above example, the port number is na.	The value can only be na.

The following is an example of changing the handling policy for the current fault from NotHandleFaultCodes (no handling required) to SeparateFaultCodes (isolate the chip and perform job rescheduling).

Log in to the environment and go to the decompression directory of Ascend Device Plugin.
Run the following command to check whether mindx-dl-fault-config has been created based on the SwitchFaultCode.json file.
```
kubectl describe cm -n kube-system mindx-dl-fault-config
```
- If mindx-dl-fault-config exists and contains the relevant fields of SwitchFaultCode.json, perform Step 4 to edit the file.
- If mindx-dl-fault-config exists but does not contain the relevant fields of SwitchFaultCode.json, save the content of mindx-dl-fault-config first, then delete the mindx-dl-fault-config file, and then perform Step 3 to create the file.
- If mindx-dl-fault-config does not exist, perform Step 3 to create the file.

Run the following command to create the mindx-dl-fault-config required for dynamic fault code configuration.

kubectl create cm mindx-dl-fault-config -n kube-system  --from-file=./faultCode.json --from-file=./SwitchFaultCode.json --from-literal="PollInterval=300"

Command output:

configmap/mindx-dl-fault-config created

Table 2 Parameter description

Parameter Name	Required	Description
mindx-dl-fault-config	Yes	The name of the ConfigMap file required for dynamic fault code configuration. This file name cannot be modified.
kube-system	Yes	The namespace where mindx-dl-fault-config resides. This namespace name cannot be modified.
SwitchFaultCode.json	Yes	Used to store fault codes. Must be consistent with the SwitchFaultCode.json file name.

Run the following command to edit the mindx-dl-fault-config file.
```
kubectl edit cm -n kube-system mindx-dl-fault-config
```

In the mindx-dl-fault-config file, locate the fault code [0x00f1ff09,155913,cpu,na].

Data
====
SwitchFaultCode.json:
----
{"NotHandleFaultCodes":[0x00f1ff09,155913,cpu,na],
...

Delete the fault code from NotHandleFaultCodes and add it to SeparateFaultCodes.

Data
====
SwitchFaultCode.json:
----
{"NotHandleFaultCodes":[],

...
"SeparateFaultCodes":["0x00f1ff09,155913,cpu,na","[0x00f103b0,155907,na,na]"…]
}

After the modification is complete, press the Esc, enter :wq! to save and exit.
After the mindx-dl-fault-config file update takes effect (PollInterval defaulted to 300s if not specified), check whether the operation is successful.
1. Run the following command to query the log name of Ascend Device Plugin.
```
kubectl get pods -A | grep ascend-device-plugin
```
  The echo example is as follows:
```
kube-system      ascend-device-plugin-daemonset-910-jmlf5   1/1     Running   0              6h34m
```
2. Query the log information of Ascend Device Plugin by using the queried component log name.
```
kubectl logs -n kube-system ascend-device-plugin-daemonset-910-jmlf5
```
  If the log displays "load switch fault code from configmap success", it indicates that the manual fault code configuration is successful.

Associated Faults

Configuration File Description

For associated faults (special faults may trigger other related faults), it is necessary to ignore the included accompanying faults. ClusterD can detect special faults and perform special processing on faulty jobs according to the associated fault policies configured in the relationFaultCustomization.json and faultDuration.json files.

relationFaultCustomization.json and faultDuration.json are system configuration files. Do not modify them arbitrarily unless you have special requirements.

Table 1 relationFaultCustomization file description

Parameter	Description	Value
TriggerFault	Accompanying fault code. Currently supports fault codes configured in faultCode.json and SwitchFaultCode.json.	String
RelationFaults	List of faults to be associated, which can be one or more fault codes. Currently supports fault codes configured in faultCode.json and SwitchFaultCode.json.	String list
FaultStrategy	Handling policy for the corresponding job when the associated fault is successfully matched. Separate: job isolation SubHealth: job sub-health	String
Note: When the configured RelationFaults occur on a device, ClusterD will add the corresponding faults to the pending fault code queue. Within the configured TimeOutInterval, if a fault corresponding to TriggerFault occurs, the job will be processed according to the user-configured FaultStrategy policy. If the configured TimeOutInterval is exceeded, bus device faults will be processed using SubHealth policy; chip faults or parameter plane network faults will be ignored.

Table 2 faultDuration.json file description

Parameter	Description	Value
FaultCode	Fault code. Currently supports fault codes configured in faultCode.json and SwitchFaultCode.json.	String
FaultType	Fault type: faultDevice: chip fault or parameter plane network fault faultSwitch: bus device fault	String
TimeOutInterval	Maximum time for which the fault code can be associated. Unit: seconds.	Integer

Parameter

Description

Value

FaultCode

Fault code. Currently supports fault codes configured in faultCode.json and SwitchFaultCode.json.

String

FaultType

Fault type:

faultDevice: chip fault or parameter plane network fault
faultSwitch: bus device fault

String

TimeOutInterval

Maximum time for which the fault code can be associated. Unit: seconds.

Integer

(Optional) Configure the Handling Policy for Associated Faults

When the ClusterD image is built, the two configuration files for associated faults are built into the image. When ClusterD starts, it reads the default configurations of these two files as the basis for current fault handling.

If you want to customize the associated fault codes and corresponding handling policies, modify the corresponding relationFaultCustomization.json and faultDuration.json files when creating the ClusterD image.

Procedure

Take RelationFaults with code 81078603 and TriggerFault with code 8C1F8609 as an example. If the fault 81078603 occurs, the fault 8C1F8609 should be ignored when it appears within the subsequent 60 seconds, and the job where the fault 81078603 occurred should be isolated. You can manually configure the handling policy for associated faults to Separate.

Log in to the environment and go to the directory where ClusterD is decompressed.
Run the vi relationFaultCustomization.json command to edit the configuration file.
```
vi relationFaultCustomization.json
```
Associate the two faults. After modification, press Esc and enter :wq! to save and exit.
```
…
  {
    "TriggerFault": "8C1F8609",
    "RelationFaults": [
      "81078603"
    ],
    "FaultStrategy": "Separate"
  }
…
```
Run the vi faultDuration.json command to edit the configuration file.
```
vi faultDuration.json
```
Configure fault types, fault association time, etc. After modification, press Esc and enter :wq! to save and exit.
```
…
  {
    "FaultCode": "81078603",
    "FaultType": "faultDevice",
    "TimeOutInterval": 60
  }
…
```

Common Faults

Configuration File Description

Resumable training performs hierarchical processing for different levels of common faults. ClusterD obtains the fault code of the current fault and processes the fault accordingly based on the fault level configured for the fault code in the publicFaultConfiguration.json file. In special cases, if ClusterD receives an unrecognized fault code (not saved in the configuration file), it will discard this fault.

publicFaultConfiguration.json is the system configuration file for common faults. Do not modify it arbitrarily unless you have special requirements. If you need to modify the level and sender of common faults, you can do so by writing a custom configuration file named publicCustomization.json to /user1/mindx-dl/clusterd. The path to this file is configurable. The configuration method is as follows:

publicCustomization.json is located at /user1/mindx-dl/clusterd inside the container. Modification and soft links are not supported. The default host path is /user1/mindx-dl/clusterd.
You can configure the host path based on actual conditions: Modify the host mount path of the volume named config-clusterd in the ClusterD startup YAML.
In a multi-master scenario, it is recommended to synchronize the latest publicCustomization.json file on each master node. This prevents the issue of losing the custom fault configuration file if ClusterD is rescheduled to another master node after a restart.

Table 1 Fault levels and handling policies

Fault Level	Fault Handling Policy	Rescheduling Handling
NotHandleFault	No handling required	Not handled for now
SeparateNPU	Unrecoverable, chip isolation required	Isolate the chip and perform job rescheduling.
SubHealthFault	Handling is based on the value of the subHealthyStrategy parameter configured in the job YAML. For details, see YAML Configuration Description.	When a chip experiences a sub-health fault, handling must follow the policy in Job YAML Configuration. If the chip subsequently experiences faults of other levels, the SubHealthFault handling policy does not affect the handling of those other-level faults.
PreSeparateNPU	No immediate service impact; subsequent jobs will no longer be scheduled to this chip.	Pre-isolate the chip.

Table 2 publicFaultConfiguration.json field description

Parameter Name	Description
publicFaultCode	Configuration related to common fault codes.
publicFaultResource	Configuration of the common fault sender.

Table 3 publicFaultCode field description

Parameter Name	Description
NotHandleFaultCodes	Fault codes with the fault level NotHandleFault (no handling required).
SubHealthFaultCodes	Fault codes with the fault level SubHealthFault.
SeparateNPUCodes	Fault codes with the fault level SeparateNPU (unrecoverable, chip isolation required).
PreSeparateNPUCodes	Fault codes with the fault level PreSeparateNPU (no immediate service impact, but no further jobs will be scheduled to this chip).

Fault Code Description

The fault code for common faults is 9 digits, as described below.

Table 4 Fault code description

Bit	Description	Value
1	Fault type	0: chip fault 1: node fault 2: network fault 3: storage fault
2	Default fault level	0: NotHandleFault 1: SubHealthFault 2: SeparateNPU
3, 4	Reserved extension bits	Temporarily 00
5	Whether the fault code in bits 6-9 is user-defined to avoid conflicts	0: defined in the release package 1: user-defined
6-9	Specific decimal fault code	Example: 1001
Examples are as follows: 0100 01001: chip fault, SubHealthFault, defined in the release package, fault 1001. 1000 11002: node fault, NotHandleFault, user-defined, fault 1002. 2200 01003: network fault, SeparateNPU, defined in the release package, fault 1003.

Known Common Faults

Table 5 Known common faults

Fault Code	Fault Description	Default Fault Level
010001001	Optical link contamination (chip fault)	SubHealthFault
210001007	Optical link contamination (network fault)	SubHealthFault
220001001	NPU card HCCS network fault	SeparateNPU
010001004	Optical link loosening (chip fault)	SubHealthFault
210001008	Optical link loosening (network fault)	SubHealthFault
310001005	DPC client failure	SubHealthFault
200001006	Suspected optical link sub-health	NotHandleFault
210001009	Optical module component sub-health	SubHealthFault
220001002	Non-existent backup rack resources used for scheduling in a back SuperPoD scenario	SeparateNPU
220001003	Resource port fault in a backup rack	SeparateNPU
220001004	Job ID occupancy conflict in a backup rack	SeparateNPU
220001005	NetMind failure	SeparateNPU
220001006	Suspected partial failure of backup rack link port	SeparateNPU
220001007	Optical link adjustment failure	SeparateNPU
200001010	Slow network generated/recovered within a node (slow network fault)	NotHandleFault
200001011	Slow network generated/recovered between nodes within a SuperPoD (slow network fault)	NotHandleFault
200001012	Slow network not caused by chip fault (slow network fault)	NotHandleFault
110001010	Slow node fault	SubHealthFault
100001011	Degradation recovered (slow node fault)	NotHandleFault
110001020	Shared storage DPC process exception	SubHealthFault
110001021	Insufficient memory for Shared storage DPC	SubHealthFault

(Optional) Configuring the Level and Sender of Common Faults

When the ClusterD image is created, the fault level configuration file publicFaultConfiguration.json is built into the image. When ClusterD starts, it reads the default configuration of this file as the basis for current fault handling.

If you want to customize fault levels, create the /user1/mindx-dl/clusterd/publicCustomization.json file on the host.

If this file exists when ClusterD starts, ClusterD will prioritize the content configured in the existing file as the basis for current fault handling.
If this file exists after ClusterD is reinstalled, the default publicFaultConfiguration.json of ClusterD will not take effect, and the existing publicCustomization.json file will be used. If you want to use the default configuration of publicFaultConfiguration.json, you can delete the existing publicCustomization.json file so that ClusterD reads the default publicFaultConfiguration.json file.
If the content of the publicCustomization.json file has issues such as format errors, ClusterD will read the content of the built-in publicFaultConfiguration.json file in the image by default as the basis for current fault handling.

Configuring the Level of Common Fault Codes

Configuring the level of common fault codes is divided into the following two scenarios.

Adjusting the level of existing fault codes.
Adding new fault codes and their fault levels.

The following uses fault code 010001008 as an example to describe how to configure a common fault code level.

Log in to the environment and go to the /user1/mindx-dl/clusterd directory.
Run the vi publicCustomization.json command to edit the file. For detailed description of publicCustomization.json, see Table 2.
- After creating the publicCustomization.json file, ensure that the file has the read permission for the ClusterD user hwMindX. For example, if the user permission is root, the file permission is recommended to be set to 644.
- Ensure the security of file permissions. Excessive permissions may pose a security risk.
```
{
  "publicFaultCode": {
    "NotHandleFaultCodes":[],
    "SubHealthFaultCodes":[],
    "SeparateNPUCodes":["010001008"],
    "PreSeparateNPUCodes":[]
  },
  "publicFaultResource": [
    "CCAE", "fd-online", "pingmesh", "Netmind", "dpcStorage"
  ]
}
```
After the modification is complete, press Esc, enter :wq! to save and exit.
After a few seconds, the file takes effect. Check whether the operation is successful.

If the log displays "load fault config from <publicCustomization.json> success", the manual fault code configuration is successful.

Configuring the Sender of Common Faults

The following uses the new fault sender XXX as an example to describe the steps for configuring the sender of common fault codes.

Run the vi publicCustomization.json command to edit the file. For detailed description of publicCustomization.json, see Table 2.

{
  "publicFaultCode": {
    "NotHandleFaultCodes":[],
    "SubHealthFaultCodes":[],
    "SeparateNPUCodes":[],
    "PreSeparateNPUCodes":[]
  },
  "publicFaultResource": [
    "CCAE", "fd-online", "pingmesh", "Netmind", "dpcStorage", "XXX"
  ]
}

After modification, press Esc and enter :wq! to save and exit.
The file takes effect after a few seconds. Check whether the operation is successful.

If "load fault config from <publicCustomization.json> success" appears in the log, the configuration is successful.