Ascend Device Plugin

Restrictions

Before installing Ascend Device Plugin, you need to understand related restrictions. For details, see Table 1.

Procedure

1. Log in to each compute node as the root user and check whether the image and version are correct.

   docker images | grep k8sdeviceplugin

   Command output:

   1	ascend-k8sdeviceplugin v7.3.0 29eec79eb693 About an hour ago 105MB

   • If correct, go to Step 2.
   • If not correct, create the image and distribute it by referring to Preparing an Image.
2. Copy the YAML file in the directory where the Ascend Device Plugin package is decompressed to any directory on the Kubernetes management node. Note that you need to use the YAML file that adapts to the specific processor model. To prevent exceptions in the automatic identification of Ascend Docker Runtime, do not modify the DaemonSet.metadata.name field in the YAML file. For details, see the following table.

   Table 2 YAML files of Ascend Device Plugin

   YAML File	Description
   device-plugin-310-v{version}.yaml	Configuration file used when Volcano is not used on an inference server (equipped with Atlas 300I inference cards).
   device-plugin-310-volcano-v{version}.yaml	Configuration file used when Volcano is used on an inference server (equipped with Atlas 300I inference cards).
   device-plugin-310P-1usoc-v{version}.yaml	Configuration file used when Volcano is not used on Atlas 200I SoC A1 core boards
   device-plugin-310P-1usoc-volcano-v{version}.yaml	Configuration file used when Volcano is used on Atlas 200I SoC A1 core boards
   device-plugin-310P-v{version}.yaml	Configuration file used when Volcano is not used on Atlas inference product
   device-plugin-310P-volcano-v{version}.yaml	Configuration file used when Volcano is used on Atlas inference product
   device-plugin-910-v{version}.yaml	Configuration file used when Volcano is not used on Atlas training product, Atlas A2 training product, Atlas A3 training product, Atlas 800I A2 inference server, or A200I A2 Box heterogeneous component
   device-plugin-volcano-v{version}.yaml	Configuration file used when Volcano is used on Atlas training product, Atlas A2 training product, Atlas A3 training product, Atlas 800I A2 inference server, or A200I A2 Box heterogeneous component

3. Skip this step if you do not need to modify the component startup parameters. Otherwise, modify the startup parameters of Ascend Device Plugin based on your requirements. For details about the startup parameters, see Table 3. You can run the ./device-plugin -h command to view the parameter descriptions.
   • On the Atlas 200I SoC A1 core board, modify the Ascend Device Plugin startup parameters in the startup script run_for_310P_1usoc.sh. After the modification, create images on all Atlas 200I SoC A1 core board nodes, or create an image on a local node and distribute the image to other Atlas 200I SoC A1 core board nodes.
     

     If Volcano is not used as the scheduler, you need to modify the Ascend Device Plugin's startup parameter in the run_for_310P_1usoc.sh file when starting Ascend Device Plugin. That is, set -volcanoType to false.

   • For other types of nodes, modify the Ascend Device Plugin's startup parameters in the corresponding startup YAML file.
4. (Optional) When resumable training (including process-level recovery) or elastic training is used, modify the startup YAML file of Ascend Device Plugin based on the fault handling mode.

   ...
         containers:
         - image: ascend-k8sdeviceplugin:v7.3.0
           name: device-plugin-01
           resources:
             requests:
               memory: 500Mi
               cpu: 500m
             limits:
               memory: 500Mi
               cpu: 500m
           command: [ "/bin/bash", "-c", "--"]
           args: [ "device-plugin  
                    -useAscendDocker=true 
                    -volcanoType=true                    # Volcano must be used in the rescheduling scenario.
                    -autoStowing=true                    # Whether to enable automatic management. The default value is true. If this parameter is set to false, automatic management is disabled. In this case, after the processor health status changes from unhealthy to healthy, or the network fault on the processor parameter plane is recovered, the processor will not be automatically added to the schedulable resource pool. This parameter applies only to Atlas training product.
                    -listWatchPeriod=5                   # Set the health status check period. The value range is [3, 1800], in seconds.
                    -hotReset=2 # When process-level recovery is used, set hotReset to 2 to enable offline recovery.
                    -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log 
                    -logLevel=0" ]
           securityContext:
             privileged: true
             readOnlyRootFilesystem: true
   ...

5. (Optional) Configure the hot reset function when recovery of inference card faults is enabled.

         containers:
         - image: ascend-k8sdeviceplugin:v7.3.0
           name: device-plugin-01
           resources:
             requests:
               memory: 500Mi
               cpu: 500m
             limits:
               memory: 500Mi
               cpu: 500m
           command: [ "/bin/bash", "-c", "--"]
           args: [ "device-plugin  
   ...
                    -hotReset=0 # Enable the hot reset function when recovery of inference card faults is enabled.
                    -logFile=/var/log/mindx-dl/devicePlugin/devicePlugin.log 
                    -logLevel=0" ]
   ...

6. (Optional) If you need to change the default port of kubelet, modify the startup YAML file of Ascend Device Plugin. Example:

     env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: KUBELET_PORT   #  Notify Ascend Device Plugin of the default kubelet port number on the current node. If the default kubelet port number is not customized, this field does not need to be passed.
          value: "10251"      
   volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
   ...

7. Run the following command, respectively, in the corresponding YAML directory on the Kubernetes management node to start Ascend Device Plugin.
   • Nodes of Atlas training product, Atlas A2 training product, Atlas A3 training product, Atlas 800I A2 inference server, or A200I A2 Box heterogeneous component exist in a Kubernetes cluster. (Volcano is used together to support virtual instances. By default, static virtualization is enabled in YAML.)

     kubectl apply -f device-plugin-volcano-v{version}.yaml

   • Nodes of Atlas training product, Atlas A2 training product, Atlas A3 training product, Atlas 800I A2 inference server, or A200I A2 Box heterogeneous component exist in a Kubernetes cluster. (Ascend Device Plugin works independently without Volcano.)

     kubectl apply -f device-plugin-910-v{version}.yaml

   • Nodes of inference servers (equipped with Atlas 300I inference cards) exist in a Kubernetes cluster. (Volcano is used together.)

     kubectl apply -f device-plugin-310-volcano-v{version}.yaml

   • Nodes of inference servers (equipped with Atlas 300I inference cards) exist in a Kubernetes cluster. (Ascend Device Plugin works independently without Volcano)

     kubectl apply -f device-plugin-310-v{version}.yaml

   • Nodes of Atlas inference product exist in a Kubernetes cluster. (Volcano is used together to support virtual instances. By default, static virtualization is enabled in YAML.)

     kubectl apply -f device-plugin-310P-volcano-v{version}.yaml

   • Nodes of Atlas inference product exist in a Kubernetes cluster. (Ascend Device Plugin works independently without Volcano.)

     kubectl apply -f device-plugin-310P-v{version}.yaml

   • Nodes of Atlas 200I SoC A1 core boards exist in a Kubernetes cluster. (Volcano is used together.)

     kubectl apply -f device-plugin-310P-1usoc-volcano-v{version}.yaml

   • Nodes of Atlas 200I SoC A1 core boards exist in a Kubernetes cluster. (Ascend Device Plugin works independently without Volcano)

     kubectl apply -f device-plugin-310P-1usoc-v{version}.yaml

   

   If multiple types of Ascend AI processors are used in a Kubernetes cluster, run the corresponding command of each type.

   Startup example:

   serviceaccount/ascend-device-plugin-sa created
   clusterrole.rbac.authorization.K8s.io/pods-node-ascend-device-plugin-role created
   clusterrolebinding.rbac.authorization.K8s.io/pods-node-ascend-device-plugin-rolebinding created
   daemonset.apps/ascend-device-plugin-daemonset created

8. Run the following command on the Kubernetes management node to check whether the component is started:

   kubectl get pod -n kube-system

   If Running is displayed in the command output, the component is started successfully.

   1 2 3 4

   NAME READY STATUS RESTARTS AGE ... ascend-device-plugin-daemonset-d5ctz 1/1 Running 0 11s ...

• After the component is installed, if the pod status of the component is not Running, refer to Component pods Are Not in the Running State.
• After the component is installed, if the pod status of the component is ContainerCreating, refer to Cluster Scheduling Component Pods Are in the ContainerCreating State.
• If the component fails to be started, refer to Cluster Scheduling Components Fail to Start and "get sem errno =13" Is Displayed in Logs.
• If the component is started successfully, but the corresponding pod cannot be found, refer to YAML File for Starting a Component Is Successfully Executed, But the pod Corresponding to the Component Is Not Displayed.

Parameters