NodeD

• NodeD must be installed when full-NPU scheduling, static vNPU scheduling, dynamic vNPU scheduling, inference card fault recovery, rescheduling upon inference card faults, resumable training, or elastic training are required.
• Users who only use containerization support and resource monitoring can skip installing NodeD and proceed directly past this chapter.
• NodeD must be installed before using the slow node & slow network fault detection feature. For details, see Slow Node & Slow Network Fault.

Procedure

1. Log in to each compute node as the root user and run the following command to check whether the image and version number are correct.

   docker images | grep noded

   The following is an example of the command output:

   noded                               v26.0.0              ef801847acd2        29 minutes ago      133MB

   • If correct, proceed to Step 2.
   • If not correct, see Preparing an Image to complete image creation and distribution.

2. Copy the YAML file in the directory where the NodeD software package is decompressed to any directory on the K8s management node.

3. If you do not modify the component startup parameters, you can skip this step. Otherwise, modify the NodeD startup parameters in the YAML file according to the actual situation. For startup parameters, see Table 1. You can run ./noded -h to view parameter descriptions.

4. (Optional) When using resumable training or elastic training, you need to configure the node status report interval. Add the -reportInterval parameter in the args line of the NodeD startup YAML file, as shown below:

   ...
             env:
               - name: NODE_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
             imagePullPolicy: Never
             command: [ "/bin/bash", "-c", "--"]
             args: [ "/usr/local/bin/noded -logFile=/var/log/mindx-dl/noded/noded.log -logLevel=0 -reportInterval=5" ]
             securityContext:
               readOnlyRootFilesystem: true
               allowPrivilegeEscalation: true
             volumeMounts:
               - name: log-noded
   ...

   • K8s marks a node as NotReady by default if no response is received within 40 seconds.
   • When the request pressure on the K8s API Server increases, you can increase the interval time according to the actual situation to reduce the load on the API Server.

5. In the path where the YAML file is located on the management node, run the following command to start NodeD.

   • If you do not use the DPC fault detection function, run the following command.

     kubectl apply -f noded-v{version}.yaml

   • If the environment requires DPC fault detection, run the following command to start NodeD.

     kubectl apply -f noded-dpc-v{version}.yaml

     Example:

     serviceaccount/noded created
     clusterrole.rbac.authorization.k8s.io/pods-noded-role created
     clusterrolebinding.rbac.authorization.k8s.io/pods-noded-rolebinding created
     daemonset.apps/noded created

6. Run the following command to check whether the component has started successfully.

   kubectl get pod -n mindx-dl

   The following is an example of the output. Running indicates that the component startup is successful.

   NAME                              READY   STATUS    RESTARTS   AGE
   ...
   noded-fd6t8                  1/1    Running  0        74s
   ...

Parameter Description

Table 1 NodeD startup parameters