ClusterD

• Users who use full-NPU scheduling, static vNPU scheduling, dynamic vNPU scheduling, resumable training, elastic training, inference card fault recovery, or rescheduling upon inference card faults must install ClusterD. ClusterD can provide full information collection services only when both Ascend Device Plugin and NodeD components exist in the cluster.
• When you install ClusterD, it is recommended to install Volcano in advance. If ClusterD is installed before Volcano, the Pod where ClusterD resides may experience CrashLoopBackOff. ClusterD will return to normal only after the Volcano Pod starts.
• Users who only use containerization support and resource monitoring do not need to install ClusterD. Please skip this chapter directly.
• Before starting ClusterD, you need to start Ascend Operator first. ClusterD needs to perceive Ascend Job resources.
• To detect slow node and slow network faults, you need to install ClusterD. For details, see Slow Node and Slow Network Faults.

Procedure

1. Log in to the K8s management node as the root user and run the following command to check whether the ClusterD image and version number are correct.

   docker images | grep clusterd

   The following is an example of the output:

   clusterd                   v26.0.0              c532e9d0889c        About an hour ago         126MB

   • If correct, perform Step 2.
   • If not correct, see Preparing an Image to complete image building and distribution.

2. Copy the YAML files from the extracted ClusterD software package directory to any directory on the K8s management node.

3. If you do not need to modify the component startup parameters, skip this step. Otherwise, modify the ClusterD startup parameters in the YAML file based on the actual situation. For startup parameters, see Table 2. You can run ./clusterd -h in the ClusterD binary package directory to view the parameter description.

4. (Optional) In clusterd-v{version}.yaml, configure the detection switch for manually isolated chips, fault frequency, de-isolation time, etc.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: clusterd-config-cm
     namespace: cluster-system
   data:
     manually_separate_policy.conf: |
       enabled: true
       separate:
         fault_window_hours: 24
         fault_threshold: 3
       release:
         fault_free_hours: 48
   

   Table 1 Parameter description of manually_separate_policy.conf

   Primary Parameter	Secondary Parameter	Type	Description
   enabled	-	bool	Detection switch for manually isolated chips. Values include: true: enabled. false: disabled. The default value is true. If this switch is turned off, all chips manually isolated by ClusterD and related caches will be cleared. [!NOTE] The YAML specification supports multiple boolean value formats (including case variants), but different parsers (such as K8s, Go, and Python) have varying compatibility, and not all formats are supported. It is recommended to use lowercase true/false uniformly.
   separate	fault_window_hours	int	Duration for manual chip isolation. Within this duration, if the number of faults with the same fault code reaches the fault_threshold value, ClusterD will manually isolate the faulty chip. The value range is [1, 720], the default value is 24, and the unit is h (hour).
   -	fault_threshold	int	Threshold for manual chip isolation. The value range is [1, 50], the default value is 3, and the unit is times.
   release	fault_free_hours	int	Unisolation time, indicating the time elapsed since the last isolation triggered by reaching the frequency threshold. After this time, the isolation will be removed. The value range is [1, 240] or -1, the default value is 48, and the unit is h (hour). The time when the frequency threshold was last reached is the LastSeparateTime in clusterd-manual-info-cm. For details about clusterd-manual-info-cm, see clusterd-manual-info-cm. Configuring -1 disables the unisolation function. When automatic unisolation is performed upon reaching the unisolation time, the isolation will be removed regardless of whether the fault has recovered.

   If the enabled field is missing, ClusterD will recognize it as false; if other int type fields are missing, ClusterD will recognize them as 0.

5. On the management node, in the path where the YAML file is located, run the following command to start ClusterD.

   kubectl apply -f clusterd-v{version}.yaml

   Example:

   clusterrolebinding.rbac.authorization.k8s.io/pods-clusterd-rolebinding created
   lease.coordination.k8s.io/cluster-info-collector created
   deployment.apps/clusterd created
   service/clusterd-grpc-svc created

6. Run the following command to check whether the component startup is successful.

   kubectl get pod -n mindx-dl

   The following is an example of the output. Running indicates that the component startup is successful.

   NAME                          READY   STATUS              RESTARTS   AGE
   clusterd-7844cb867d-fwcj7     0/1     Running            0          45s

Parameter Description

Table 2 ClusterD startup parameters