Ascend Operator

• Users who use full-NPU scheduling (training), static vNPU scheduling (training), resumable training, or elastic training must install Ascend Operator. If Volcano is used as the scheduler, it must be installed first; otherwise, Ascend Operator will fail to start.
• Users who use full-NPU scheduling (inference) and scheduling upon inference card faults to submit distributed inference job of the AscendJob type must install Ascend Operator.
• Users who only use containerization and resource monitoring, inference card fault recovery, or scheduling upon inference card faults (single-node jobs) do not need to install Ascend Operator. Please skip this chapter directly.
• Before starting Ascend Operator, Volcano must be started first. Ascend Operator needs to perceive PodGroup resources.
• The maximum number of replicas for a single AscendJob that Ascend Operator allows to create is 20,000.

Procedure

1. Log in to the K8s management node as the root user and run the following command to check whether the Ascend Operator image and version number are correct.

   docker images | grep ascend-operator

   The following is an example of the output:

   ascend-operator                      v26.0.0              c532e9d0889c        About an hour ago         137MB

   • If correct, proceed to Step 2.
   • If not correct, see Preparing Images to complete image creation and distribution.

2. Copy the YAML files from the extracted Ascend Operator software package directory to any directory on the K8s management node.

3. If you do not modify the component startup parameters, you can skip this step. Otherwise, modify the Ascend Operator startup parameters in the YAML file based on the actual situation. For startup parameters, see Table 1. You can run ./ascend-operator -h to view parameter descriptions.

4. (Optional) Use Ascend Operator to generate the collective communication configuration file (RankTable File, also known as the hccl.json) for training jobs under the PyTorch and MindSpore frameworks, to shorten the cluster communication link establishment time. Users of other frameworks can skip this step.

   1. The parent directory of the hccl.json file is already mounted by default in the startup YAML. Modify it based on the actual situation.

       ...
               - name: ranktable-dir
                 mountPath: /user/mindx-dl/ranktable        # Container path, cannot be modified
       ...
             volumes:
               - name: ascend-operator-log
                 hostPath:
                   path: /var/log/mindx-dl/ascend-operator
                   type: Directory
               - name: ranktable-dir
                 hostPath:
                   path: /user/mindx-dl/ranktable    # Host path. The root directory of the hccl.json file save path in the job YAML must be consistent with the host path.
                   type: DirectoryOrCreate                                      # Used to check whether the given folder exists. If it does not exist, an empty folder will be created.
       ...

      • The root directory path of the RankTable inside the container cannot be modified, but the host path can be modified. When deploying a job, the root directory of the hccl.json file save path in the job YAML must be consistent with the host path.
      • The permissions of the RankTable root directory folder must meet one of the following conditions:
        • The owning user and user group are hwMindX (the default running user of cluster scheduling components).
        • The permissions of the RankTable root directory folder are 777.

   2. Run the following command to create the specific mount path for the hccl.json file in the parent directory.

      mkdir -m 777 /user/mindx-dl/ranktable/{Mount path}

5. In the path where the YAML file is located on the management node, run the following command to start Ascend Operator.

   kubectl apply -f ascend-operator-v{version}.yaml

   The following is a startup example:

   deployment.apps/ascend-operator-manager created
   serviceaccount/ascend-operator-manager created
   clusterrole.rbac.authorization.k8s.io/ascend-operator-manager-role created
   clusterrolebinding.rbac.authorization.k8s.io/ascend-operator-manager-rolebinding created
   customresourcedefinition.apiextensions.k8s.io/ascendjobs.mindxdl.gitee.com created
   ...

6. Run the following command to check whether the component has started successfully.

   kubectl get pod -n mindx-dl

   The following is an example of the output. Running indicates that the component startup is successful.

   NAME                                         READY   STATUS    RESTARTS   AGE
   ...
   ascend-operator-7667495b6b-hwmjw      1/1    Running  0         11s

Parameter Description

Table 1 Ascend Operator startup parameters