NodeD

Procedure

1. Log in to each compute node as the root user and check whether the image and version are correct.

   docker images | grep noded

   Command output:

   1	noded v7.3.0 ef801847acd2 29 minutes ago 133MB

   • If correct, proceed to Step 2.
   • If not correct, create the image and distribute it by referring to Preparing an Image.
2. Copy the YAML file in the directory where the NodeD package is decompressed to any directory on the Kubernetes management node.
3. Skip this step if you do not need to modify the component startup parameters. Otherwise, modify the NodeD startup parameters in the YAML file based on your requirements. For details about the startup parameters, see Table 1. You can run the ./noded -h command to view the parameter descriptions.
4. (Optional) If resumable training or elastic training is used, configure the interval for reporting the node status. Add the -reportInterval parameter to the args line in the NodeD startup YAML file as follows:

   ...
             env:
               - name: NODE_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
             imagePullPolicy: Never
             command: [ "/bin/bash", "-c", "--"]
             args: [ "/usr/local/bin/noded -logFile=/var/log/mindx-dl/noded/noded.log -logLevel=0 -reportInterval=5" ]
             securityContext:
               readOnlyRootFilesystem: true
               allowPrivilegeEscalation: true
             volumeMounts:
               - name: log-noded
   ...

   

   • If no response is received from the node within 40 seconds by default, Kubernetes sets the node status to NotReady.
   • When the request pressure of Kubernetes API server increases, increase the interval based on the actual situation to reduce the API server stress.
5. Run the following command in the directory where the YAML file of the management node is stored to start NodeD.
   • If the DPC fault detection function is not used, run the following command:

     kubectl apply -f noded-v{version}.yaml

   • If Scale-Out Storage DPC 24.2.0 or later has been deployed in the environment and the DPC fault detection function is used, run the following command to start NodeD:

     kubectl apply -f noded-dpc-v{version}.yaml

     Startup example:

     serviceaccount/noded created
     clusterrole.rbac.authorization.k8s.io/pods-noded-role created
     clusterrolebinding.rbac.authorization.k8s.io/pods-noded-rolebinding created
     daemonset.apps/noded created

6. Run the following command to check whether the component is started successfully:

   kubectl get pod -n mindx-dl

   The following is a startup example. If Running is displayed, the component is started successfully.

   1 2 3 4

   NAME READY STATUS RESTARTS AGE ... noded-fd6t8 1/1 Running 0 74s ...

• After the component is installed, if the pod status of the component is not Running, refer to Component pods Are Not in the Running State.
• After the component is installed, if the pod status of the component is ContainerCreating, refer to Cluster Scheduling Component Pods Are in the ContainerCreating State.
• If the component fails to be started, refer to Cluster Scheduling Components Fail to Start and "get sem errno =13" Is Displayed in Logs.
• If the component is started successfully, but the corresponding pod cannot be found, refer to YAML File for Starting a Component Is Successfully Executed, But the pod Corresponding to the Component Is Not Displayed.

Parameters