Public Fault APIs

ConfigMap

Description

Receives ConfigMap information for public faults and accesses the resumable training process.

If the parameters in the actual ConfigMap do not match the defined value ranges, ClusterD will discard the fault information and ignore it.
For public faults injected through the ConfigMap or gRPC interface, the total number of faults across all nodes is capped at 50,000. When the number of faults exceeds 50,000, any further fault injection will cause ClusterD to discard the fault information and ignore it.
The label of the ConfigMap must be mc-consumer-publicfault=true, and the key of Data must be PublicFault.
When sending public faults through the ConfigMap method, ensure that the data volume of a single request cannot exceed 1 MB; otherwise, the ConfigMap update will fail.

Parameters

For details about the parameters, see the following table.

Table 1 Fault information description

Parameter Name	Description	Value	Type	Mandatory
id	Unique message identifier	A string of 8 to 128 characters, supporting uppercase and lowercase letters, digits, hyphens (-), underscores (_), and dots (.), and must be unique.	string	Yes
timestamp	Timestamp when the message is sent	Timestamp (unit: ms), 13 digits, must be after 2025-01-01T00:00:00Z.	int64	Yes
version	Message version number	Value: 1.0.	string	Yes
resource	Fault sender	Default configuration: CCAE, fd-online, pingmesh, Netmind, dpcStorage. The fault sender for public faults must exist in the publicFaultResource of the fault configuration file. For a new fault sender, you need to manually configure it in the fault configuration file. For details, see (Optional) Configuring Public Fault Levels and Senders.	string	Yes
faults	Fault content	Slice, length >0 and ≤100.	[]object, fault	Yes

Table 2 fault field description

Parameter Name	Description	Value	Type	Mandatory
faultId	Fault instance ID	A string of 8 to 128 characters, supporting uppercase and lowercase letters, digits, hyphens (-), underscores (_), and dots (.). Must be unique. For the same fault instance, the faultId must be unique.	string	Yes
faultType	Fault type	Value: `NPU`: NPU fault. `Node`: Node fault. `Network`: Network fault. `Storage`: Storage fault. This field is displayed as `"PublicFault"` in `cluster-info-cm`.	string	Yes
faultCode	Fault code	User-definable, must be a unique 9-digit code. Fault codes for accessing resumable training must exist in the `publicFaultCode` of the fault configuration file. For new fault codes, configure their fault levels in the fault configuration file. For details, see (Optional) Configuring Public Fault Levels and Senders. It is recommended that fault codes follow the rules defined in the fault code description table for easier maintenance. If the same fault code occurs twice on an NPU, the `fault_code` field in `cluster-info-cm` will record two identical fault codes.	string	Yes
faultTime	Fault generation time	Timestamp (unit: ms), 13 digits, must be after 2025-01-01T00:00:00Z. Whether it is fault generation or fault recovery, this field is always the fault generation time. This field is displayed in seconds in `cluster-info-cm`.	int64	Yes
assertion	Fault status	Value: `occur`: Fault generation. `recover`: Fault recovery. `once`: One-time event. NOTE Public fault recovery requires writing the recover event of the corresponding fault into the ConfigMap, and cannot be achieved by deleting the ConfigMap. For one-time events, the fault is automatically cleared after a few seconds.	string	Yes
faultLocation	Fault location information	Fault source information, length ≤10, map key length ≤16, value length ≤128. e.g. `key: npuIp, value: ip`	map[string]string	No
influence	Scope of fault impact	Slice, length >0 and ≤1000.	[]object, faultInfo	Yes
description	Fault description	0 to 512 characters. Contains non-whitespace characters and spaces.	string	No

Table 3 faultInfo description

Parameter Name	Description	Value	Type	Mandatory
nodeName	Node Name. Can be queried using the kubectl get nodes -owide command.	A string of 1 to 253 characters, supporting lowercase letters, digits, hyphens (-), and dots (.), and must start and end with an alphanumeric character. When this field is present, `nodeSN` is not used. If the node name does not exist in the K8s cluster, ClusterD will not report a node name error, nor write this fault information to `cluster-info-device-cm`.	string	Either `nodeName` or `nodeSN` is required
nodeSN	Node SN	The SN of the node. The value is the node annotation written by NodeD, with the key `product-serial-number`. If this field is used instead of nodeName, NodeD must be installed in advance.	string	Either `nodeName` or `nodeSN` is required
deviceIds	NPU Physical ID	Length (0, 32], each element value [0, 32), and duplicates are not allowed. If the faulty NPU cannot be accurately identified, all NPU physical IDs on the node must be filled in. If a non-existent NPU physical ID on the node is passed in, ClusterD will still display it in `cluster-info-device-cm`.	[]int32	Yes

gRPC Interface

Function Description

Receives and processes public fault sending requests from gRPC clients, and accesses the resumable training process.

If the actual gRPC request parameters do not conform to the defined value ranges, ClusterD will discard the fault information and ignore it.
For public faults injected via ConfigMap or gRPC interface, the total number of faults across all nodes is capped at 50,000. When the number of faults exceeds 50,000, any further fault injection will cause ClusterD to discard the fault information and ignore it.
To clear a public fault, the recover event of the corresponding fault must be sent to ClusterD via the gRPC interface.

Prototype

rpc SendPublicFault(PublicFaultRequest) returns (RespStatus){}

Input Parameters

Parameter	Type (Protobuf Definition)	Description
PublicFaultRequest	message PublicFaultRequest{ string id = 1; int64 timestamp = 2; string version = 3; string resource = 4; repeated Fault faults = 5; } message Fault{ string faultId = 1; string faultType = 2; string faultCode = 3; int64 faultTime = 4; string assertion = 5; map<string, string> faultLocation = 6; repeated PubFaultInfo influence = 7; string description = 8; } message PubFaultInfo{ string nodeName = 1; string nodeSN = 2; repeated int32 deviceIds = 3; }	PublicFaultRequest.id: Unique message identifier PublicFaultRequest.timestamp: Timestamp when the message is sent PublicFaultRequest.version: Message version number PublicFaultRequest.resource: Fault sender PublicFaultRequest.faults: Fault content Fault.faultId: Fault instance ID Fault.faultType: Fault type Fault.faultCode: Fault code Fault.faultTime: Fault generation time Fault.assertion: Fault status Fault.faultLocation: Fault location information Fault.influence: Scope of fault impact Fault.description: Fault description PubFaultInfo.nodeName: Node name PubFaultInfo.nodeSN: Node SN PubFaultInfo.deviceIds: NPU physical ID For details about the above parameters and their values, see ConfigMap.

Parameter

Type (Protobuf Definition)

Description

PublicFaultRequest

message PublicFaultRequest{

string id = 1;

int64 timestamp = 2;

string version = 3;

string resource = 4;

repeated Fault faults = 5;

}

message Fault{

string faultId = 1;

string faultType = 2;

string faultCode = 3;

int64 faultTime = 4;

string assertion = 5;

map<string, string> faultLocation = 6;

repeated PubFaultInfo influence = 7;

string description = 8;

}

message PubFaultInfo{

string nodeName = 1;

string nodeSN = 2;

repeated int32 deviceIds = 3;

}

PublicFaultRequest.id: Unique message identifier

PublicFaultRequest.timestamp: Timestamp when the message is sent

PublicFaultRequest.version: Message version number

PublicFaultRequest.resource: Fault sender

PublicFaultRequest.faults: Fault content

Fault.faultId: Fault instance ID

Fault.faultType: Fault type

Fault.faultCode: Fault code

Fault.faultTime: Fault generation time

Fault.assertion: Fault status

Fault.faultLocation: Fault location information

Fault.influence: Scope of fault impact

Fault.description: Fault description

PubFaultInfo.nodeName: Node name

PubFaultInfo.nodeSN: Node SN

PubFaultInfo.deviceIds: NPU physical ID

For details about the above parameters and their values, see ConfigMap.

Return Value

Return Value Type (Protobuf Definition) Description

RespStatus

Return Value	Type (Protobuf Definition)	Description
RespStatus	message RespStatus{ int32 code = 1; string info = 2; }	RespStatus.code: Return code. `0`: Fault sent successfully. Other values: Fault sending failed. `409` indicates invalid request parameters, and `410` indicates the message sending frequency exceeds the limit. RespStatus.info: Return information.

message RespStatus{

int32 code = 1;

string info = 2;

}

RespStatus.code: Return code.

0: Fault sent successfully.
Other values: Fault sending failed. 409 indicates invalid request parameters, and 410 indicates the message sending frequency exceeds the limit.

RespStatus.info: Return information.