K8s Native Object Description

Service Labels

Table 1 Service labels used by cluster scheduling components

Name	Function	Value	Component
group-name	Marks the group name of the acjob corresponding to the Pod	mindxdl.gitee.com	Volcano, Ascend Operator
job-name	Marks the acjob name corresponding to the Pod	String	Ascend Operator
replica-index	Marks the Pod sequence number (to be deleted later)	[0-{Pod quantity-1}]	Ascend Operator
replica-type	Marks the Pod type (to be deleted later)	master chief scheduler worker	Ascend Operator
training.kubeflow.org/job-name	Marks the acjob name corresponding to the Pod	String	Ascend Operator
training.kubeflow.org/operator-name	Marks the operator name that created the Pod	ascendjob-controller	Ascend Operator
training.kubeflow.org/replica-index	Marks the Pod sequence number	[0-{Pod quantity-1}]	Ascend Operator
training.kubeflow.org/replica-type	Marks the Pod type	master chief scheduler worker	Ascend Operator

Job Labels

Table 2 Job labels used by cluster scheduling components

Job Label	Function	Value	Component
mind-cluster/scaling-rule: scaling-rule	Marks the ConfigMap name corresponding to the scaling rule.	String	Ascend Operator
mind-cluster/group-name: group0	Marks the group name corresponding to the scaling rule.	String	Ascend Operator

Job Annotations

Table 3 Job annotations used by cluster scheduling components

Job Annotation	Function	Value	Component
huawei.com/schedule.filter.faultCode	Configures the fault codes and time windows that need to be silenced for the configuration task. Fault codes only support configuring fault codes for chip faults and UnifiedBus device faults. For details on supported fault codes, see the faultCode.json and SwitchFaultCode.json files. Supports configuring multiple fault codes and time windows, separated by commas.	Value example: "8C1F8608:30, 80E01801", indicating that within a 30-second time window, the 8C1F8608 fault is silenced; within a 60-second time window, the 80E01801 fault is silenced. If no time window is configured, the default is 60. The value range is 0 to 86400, in seconds.	ClusterD
huawei.com/schedule.filter.faultLevel	Configures the fault levels and time windows that need to be silenced for the configuration task. Fault levels only support configuring levels for chip faults and UnifiedBus device faults. For details on supported fault levels, see Configuration Description. Supports configuring multiple fault levels and time windows, separated by commas. For MindIE Service, if this configuration item is absent in the YAML file, all RestartRequest-level faults are silenced for 60 seconds by default. The priority of huawei.com/schedule.filter.faultCode is higher than that of huawei.com/schedule.filter.faultLevel. For notification-type faults, after ClusterD silences such faults, Volcano may not actively reschedule the faulty Pod. The task can subscribe to ClusterD's fault subscription interface to handle received faults accordingly. If processing fails, the Pod must actively exit with an error.	Value example: "RestartRequest:30, RestartBusiness", indicating that within a 30-second time window, all RestartRequest-level faults are silenced; within a 60-second time window, all RestartBusiness-level faults are silenced. If no time window is configured, the default is 60. The value range is 0 to 86400, in seconds.	ClusterD

Job Annotation

Function

Value

Component

huawei.com/schedule.filter.faultCode

Configures the fault codes and time windows that need to be silenced for the configuration task.

Fault codes only support configuring fault codes for chip faults and UnifiedBus device faults. For details on supported fault codes, see the faultCode.json and SwitchFaultCode.json files.
Supports configuring multiple fault codes and time windows, separated by commas.

Value example: "8C1F8608:30, 80E01801", indicating that within a 30-second time window, the 8C1F8608 fault is silenced; within a 60-second time window, the 80E01801 fault is silenced.

If no time window is configured, the default is 60. The value range is 0 to 86400, in seconds.

ClusterD

huawei.com/schedule.filter.faultLevel

Configures the fault levels and time windows that need to be silenced for the configuration task.

Fault levels only support configuring levels for chip faults and UnifiedBus device faults. For details on supported fault levels, see Configuration Description.
Supports configuring multiple fault levels and time windows, separated by commas.
For MindIE Service, if this configuration item is absent in the YAML file, all RestartRequest-level faults are silenced for 60 seconds by default.
The priority of huawei.com/schedule.filter.faultCode is higher than that of huawei.com/schedule.filter.faultLevel.
For notification-type faults, after ClusterD silences such faults, Volcano may not actively reschedule the faulty Pod. The task can subscribe to ClusterD's fault subscription interface to handle received faults accordingly. If processing fails, the Pod must actively exit with an error.

Value example: "RestartRequest:30, RestartBusiness", indicating that within a 30-second time window, all RestartRequest-level faults are silenced; within a 60-second time window, all RestartBusiness-level faults are silenced.

If no time window is configured, the default is 60. The value range is 0 to 86400, in seconds.

ClusterD

Node Labels

Table 4 Node labels used by cluster scheduling components

Node Label	Function	Value	Component
accelerator	Automatically labeled by Ascend Device Plugin; no manual configuration required by the user.	huawei-npu huawei-Ascend910 huawei-Ascend310 huawei-Ascend310P	Ascend Device Plugin
masterselector	Identifies the management node of MindCluster.	dls-master-node	Volcano, Ascend Operator, Resilience Controller, ClusterD
node.kubernetes.io/npu.chip.name	Reports the specific type of the current chip.	310 310P1 310P2 310P3 310P4 {xxx}A 910PremiumA 910ProA 910ProB {xxx}Bx(where x can be 1, 2, 3, or 4) Ascend950PR Ascend950DT	Ascend Device Plugin [!NOTE] {xxx} takes "910" as the chip model number.
workerselector	Identifies the compute node of MindCluster.	dls-worker-node	Ascend Device Plugin, NodeD, NPU Exporter
servertype	Device type	npu-{aicore_core_count} soc Ascend910-{aicore_core_count} Ascend310P-{aicore_core_count}	Volcano, Ascend Device Plugin
huawei.com/driver.version	HDK driver version number	The corresponding HDK version number of the node.	Ascend Device Plugin
huawei.com/Ascend910-Recover huawei.com/npu-Recover	Fault recovery identifier for Atlas training products	Faulty chip ID	Ascend Device Plugin
huawei.com/Ascend910-NetworkRecover huawei.com/npu-NetworkRecover	Network fault recovery identifier for Atlas training products	Faulty chip ID	Ascend Device Plugin
infer-card-type	Written by Ascend Device Plugin, indicating the inference card type of the node.	card-300i-duo	Volcano
mind-cluster/npu-chip-memory	On-chip memory of the chip	64G	Volcano, Ascend Device Plugin
huawei.com/scheduler.chip1softsharedev.enable	Indicates whether the node supports the soft-partitioning virtualization feature.	true false	Volcano, Ascend Device Plugin [!NOTE] huawei.com/scheduler.chip1softsharedev.enable=true indicates that the node supports the soft-partitioning virtualization feature. huawei.com/scheduler.chip1softsharedev.enable=false indicates that the node does not support the soft-partitioning virtualization feature.
huawei.com/topotree.rackid	Identifies the rack ID of the node.	The rack ID to which the node belongs.	Volcano
huawei.com/topotree.serverid	Identifies the index ID of the node within the SuperPoD.	The index ID within the SuperPoD to which the node belongs	Volcano
huawei.com/topotree.superpodid	Identifies the SuperPoD ID of the node.	The SuperPoD ID to which the node belongs.	Volcano
huawei.com/topotree.groupid	Identifies the Pod group ID of the node.	The Pod group ID to which the node belongs.	Volcano
huawei.com/topotree	Identifies the network topology tree ID of the node.	The network topology tree ID to which the node belongs.	Volcano

Pod Labels

Table 5 Pod Labels used by the cluster scheduling components

Name	Function	Value	Component
ring-controller.atlas	Identify Atlas Pod	ascend-910 ascend-{xxx}b ascend-npu	Ascend Device Plugin
vnpu-dvpp	Mark the DVPP set for the Pod	yes: This Pod uses DVPP. no: This Pod does not use DVPP. null: Default value. Does not care whether DVPP is used.	Volcano
vnpu-level	Mark the level of the selected virtualization instance template	low: Low configuration, default value. high: Performance priority.	Volcano
version	Mark the version of the Pod	String	Ascend Operator
volcano.sh/job-name	Mark the vcjob name corresponding to the Pod	String	Volcano
volcano.sh/job-namespace	Mark the vcjob namespace corresponding to the Pod	String	Volcano
volcano.sh/queue-name	Mark the queue name corresponding to the Pod	String	Volcano
volcano.sh/task-spec	Mark the job name corresponding to the Pod	String	Volcano
fault-type	Mark the Pod fault handling policy	SubHealth Separate	Volcano
deploy-name	Mark the deployment name corresponding to the Pod	String	Ascend Operator
group-name	Mark the group name of the acjob corresponding to the Pod	mindxdl.gitee.com	Volcano, Ascend Operator
job-name	Mark the acjob name corresponding to the Pod	String	Ascend Operator
replica-index	Mark the Pod index (to be deleted later)	[0-{Pod Quantity-1}]	Ascend Operator
replica-type	Mark the Pod Type (to be deleted later)	master chief scheduler worker	Ascend Operator
training.kubeflow.org/job-name	Mark the acjob name corresponding to the Pod	String	Ascend Operator
training.kubeflow.org/job-role	Mark the Pod Type	master	Ascend Operator
training.kubeflow.org/operator-name	Mark the operator name that created the Pod	ascendjob-controller	Ascend Operator
training.kubeflow.org/replica-index	Mark the Pod index	[0-{Pod Quantity-1}]	Ascend Operator
training.kubeflow.org/replica-type	Mark the Pod type	master chief scheduler worker	Ascend Operator
super-pod-affinity	Affinity scheduling policy used by SuperPoD jobs	soft hard	Ascend Operator, Volcano

Pod Annotations

Table 6 Pod Annotations used by the cluster scheduling components

Name	Function	Value	Component
ascend.kubectl.kubernetes.io/ascend-910-configuration ascend.kubectl.kubernetes.io/ascend-npu-configuration	Data source for Ascend Operator to generate hccl.json	String map	Ascend Device Plugin, Ascend Operator
super_pod_id	Provides SuperPoD ID information for Ascend Operator	Number	Ascend Operator
hccl/rankIndex	Basis for retaining the original rank ID during resumable training	[0,1000]	Volcano, Ascend Operator
distributed-job	Marks the training job type	true: The current job is a distributed job false: The current job is a single-server job	Volcano
huawei.com/Ascend910 huawei.com/npu	Basis for Ascend Device Plugin to allocate chips to Pods. The specific value corresponds to davinciX under /dev/. For the Ascend950 generation, it is the NPU ID; for previous generations, it is the physical ID.	String	Volcano, Ascend Device Plugin
huawei.com/AscendReal	Record of the actual chips allocated by Ascend Device Plugin to the Pod	String	Volcano, Ascend Device Plugin
huawei.com/npu-core	Marks the physical ID and slicing template of the NPU card used by the Pod	String	Volcano, Ascend Device Plugin
huawei.com/kltDev	Record of chips allocated by kubelet to the Pod	String	Ascend Device Plugin
huawei.com/recover_policy_path	Job rescheduling policy	pod: Only supports Pod-level rescheduling, will not escalate to Job level (when using vcjob, this policy needs to be configured: policies: -event:PodFailed -action:RestartTask)	Volcano
huawei.com/schedule_minAvailable	Minimum number of replicas required for the job to be scheduled	Integer	Volcano
predicate-time	Basis for the order in which Ascend Device Plugin allocates chips to Pods	String	Volcano, Ascend Device Plugin
isSharedTor	Marks the switch attributes corresponding to the Pod	Integer	Volcano
isHealthy	Marks the switch status corresponding to the Pod	Integer	Volcano
scheduling.k8s.io/group-name	Marks the podGroup name corresponding to the Pod	String	Volcano
volcano.sh/job-name	Marks the vcjob name corresponding to the Pod	String	Volcano
volcano.sh/job-version	Marks the vcjob version corresponding to the Pod	String	Volcano
volcano.sh/queue-name	Marks the queue name corresponding to the Pod	String	Volcano
volcano.sh/task-spec	Marks the job name corresponding to the Pod	String	Volcano
volcano.sh/template-uid	Marks the pod-template name corresponding to the Pod	String	Volcano
sharedTorIp	Marks the shared switch information used by the job	String	Volcano, ClusterD
fault-job-delete	Marks the rank information of the job	String	Volcano
mind-cluster/hardware-type=800I-A2-xx	xx indicates the on-chip memory of the current node, for example, mind-cluster/hardware-type=800I-A2-64G	String	Volcano
super-pod-rank	Logical SuperPoD rank of the job	Number	Ascend Operator, Volcano
inHotSwitchFlow	Marks that the current Pod (faulty Pod and backup Pod) is in a hot switching process	true	ClusterD, Ascend Operator
backupNewPodName	Marks the name of the backup Pod created for the current faulty Pod	Corresponding backup Pod name	ClusterD, Ascend Operator
backupSourcePodName	Marks the original Pod name corresponding to the current backup Pod	Corresponding original Pod name	Ascend Operator
needOperatorOpe	Marks that the current Pod needs to be processed by Ascend Operator	create: Ascend Operator needs to create a backup Pod based on the current Pod delete: Ascend Operator needs to delete the current Pod	ClusterD, Ascend Operator
needVolcanoOpe	Marks that the current Pod needs to be processed by Volcano	delete: Volcano needs to delete the current Pod	ClusterD, Volcano
podType	Marks that the current Pod is a backup Pod	backup	ClusterD, Ascend Operator
huawei.com/scheduler.softShareDev.aicoreQuota	Marks the percentage of AICore required by the current Pod.	[1, 100]	Volcano, Ascend Device Plugin
huawei.com/scheduler.softShareDev.hbmQuota	Marks the amount of high-bandwidth memory required by the current Pod.	[1, maxHBM] maxHBM is the HBM value in HBM-Usage(MB) queried using the npu-smi info command.	Volcano, Ascend Device Plugin
huawei.com/scheduler.softShareDev.policy	Marks the policy of the soft partitioning job executed by the current Pod.	fixed-share elastic best-effort	Volcano, Ascend Device Plugin
huawei.com/affinity-config	Configures the affinity level for multi-level scheduling of the job.	level1=x,level2=y,... Where x, y... are the sub-job sizes for the corresponding network levels. This field is used to configure the affinity level for multi-level scheduling of the job. It must be a concatenation of strings in the format leveli=ni, separated by commas. Here, i is the network level sequence number, and ni is the number of replicas for the sub-job at that network level. For example, for a job with a total of 8 replicas, "level1=2,level2=4" means that every 2 Pods in the job are assigned to nodes with the same level1 label, and every 4 Pods are assigned to nodes with the same level2 label. The network level configuration must meet the following requirements: When the job has more than one level, the value of level n must be an integer multiple of n-1. The total number of job replicas must be an integer multiple of all levels. The job level configuration must start from level1 and be consecutive in ascending order.	Volcano
huawei.com/schedule_policy	Specifies the scheduling policy.	Currently supports the configurations in Table 3 huawei.com/schedule_policy Configuration Description.	Volcano
volcano.sh/schedule.plugin	When NPU resources are not configured in MinResources, this annotation is used to declare the NPU type that the job uses, so that Volcano still schedules the job according to NPU affinity scheduling rules.	The value is NPU type, for example, Ascend910, Ascend310P, and Ascend310.	Volcano

Node Annotations

Table 7 Node annotations used by the cluster scheduling components

Name	Function	Value	Component
baseDeviceInfos	Displays basic chip information, such as IP, for use during Volcano scheduling.	String	Volcano
product-serial-number	NodeD obtains the node SN through the IPMI and writes it into the annotation for use when ClusterD receives a public fault.	String	ClusterD
superPodID	Indicates the ID of the SuperPoD to which this node belongs.	String	ClusterD
ResetInfo	Displays information about chips that failed automatic reset by the Ascend Device Plugin, such as the chip's physical ID, Card ID, etc.	String	Ascend Device Plugin

The content format of ResetInfo is as follows.

{
    "ThirdPartyResetDevs": [
        {
            "CardId": 0,
            "DeviceId": 0,
            "AssociatedCardId": 4,
            "PhyID": 0,
            "LogicID": 0
        }
    ],
    "ManualResetDevs": [
        {
            "CardId": 1,
            "DeviceId": 0,
            "AssociatedCardId": 5,
            "PhyID": 2,
            "LogicID": 2
        }
    ]
}

K8s ServiceAccount

Table 8 List of ServiceAccounts created by components in K8s

Account Name	Description
volcano-controllers	User created in K8s by the controller component of open-source Volcano.
volcano-scheduler	User created in K8s by the scheduler component of open-source Volcano.
ascend-device-plugin-sa-npu ascend-device-plugin-sa-910 ascend-device-plugin-sa-310p ascend-device-plugin-sa-310	When starting the service using YAML, this user will be created in K8s. The Account Name used varies for different device models.
ascend-operator-manager	When starting the service using YAML, this user will be created in K8s, for example: ascend-operator-v{version}.yaml.
resilience-controller	It is recommended to start with security hardening. Use the YAML with `without-token` to start the service, create and use the resilience-controller account in K8s, and grant appropriate permissions to this account.
noded	When starting the service using YAML, this user will be created in K8s, for example: noded-v{version}.yaml.
clusterd	When starting the service using YAML, this user will be created in K8s, for example: clusterd-v{version}.yaml.
default	User automatically created in K8s when deploying MindCluster components or open-source Volcano.