Deploying MindIE Motor

Implementation Principle

The steps are described as follows:

1. The cluster scheduling components periodically report node and chip information.

   • kubelet reports the number of node chips to the node object.

   • Ascend Device Plugin reports chip memory and topology information.

     For chips with on-chip memory, Ascend Device Plugin reports the chip memory status upon startup, as described in the node-label description; reports the full NPU information, writing the physical ID of the chip to device-info-cm; the total number of allocatable le chips, the number of allocated chips, and basic chip information (device ip and super_device_ip) are reported to the node for full NPU scheduling.

   • When a fault exists on a node, NodeD periodically reports the node health status and node hardware fault information to node-info-cm, and reports shared storage fault information to the public faults of ClusterD.

2. After reading the information in device-info-cm, node-info-cm, and the public faults, ClusterD writes the information into cluster-info-cm.

3. The user submits MS Controller, MS Coordinator tasks that do not use NPUs, and several MindIE Server tasks that use NPUs, through kubectl or other deep learning platforms.

4. Ascend Operator creates the corresponding podGroup for the tasks. For detailed information about podGroup, refer to the official open-source Volcano documentation.

5. Ascend Operator creates the corresponding Pods for the tasks and injects the environment variables required for starting the MindIE Server service. For detailed information about environment variables, see Environment Variables Injected by Ascend Operator.

6. For MS Controller and MS Coordinator tasks, volcano-scheduler selects appropriate nodes based on node memory, CPU, labels, and affinity. For MindIE Server tasks, volcano-scheduler also references chip topology information to select appropriate nodes, and writes the selected chip information and node hardware information into the Pod's annotations.

7. When kubelet creates containers, for MindIE Server tasks, it calls Ascend Device Plugin to mount chips. Ascend Device Plugin or volcano-scheduler writes chip and node hardware information into the Pod's annotations. Ascend Docker Runtime assists in mounting the corresponding resources.

8. Ascend Operator reads the annotation information of each MindIE Server task Pod, generates their respective collective communication file hccl.json, and stores it in etcd in the form of a ConfigMap.

9. ClusterD listens for changes in MS Controller and MS Coordinator task Pod information, as well as changes in the ConfigMaps corresponding to each hccl.json, and generates the global-ranktable in real time. For a detailed description of the global-ranktable, see the "global-ranktable File Description" section in SubscribeRankTable.

10. After MS Controller starts, it establishes communication with ClusterD and subscribes to changes in the global-ranktable through the gRPC interface.

Using via Command Line

Process Description

MindIE Motor consists of two parts: MindIE MS (MindIE Management Service) and MindIE Server. MindIE MS includes MS Controller and MS Coordinator, while MindIE Server can be divided into Prefill instances and Decode instances. MS Controller and MS Coordinator do not require NPU resources, whereas MindIE Server requires NPU resources.

The MindCluster cluster scheduling components support running MS Controller, MS Coordinator, and MindIE Server in independent Pods. When using the MindCluster cluster scheduling components for MindIE Motor task deployment, MS Controller, MS Coordinator, and each instance in MindIE Server are deployed as an AscendJob respectively. For example, if an inference job contains two Prefill instances and one Decode instance, five AscendJobs need to be deployed.

For detailed instructions on Prefill-Decode disaggregation deployment, refer to the "Cluster Service Deployment > Prefill-Decode Disaggregation Deployment" section in the MindIE Motor Development Guide.

Usage Process

The figure below shows the process of deploying MindIE Motor inference jobs using the MindCluster cluster scheduling components via the command line.

Figure 1 Usage process

Preparing a Job YAML

Prepare for image creation as required, select a YAML file, and modify the YAML file.

Prerequisites

The image preparation has been completed.

YAML Selection

Various YAML examples are provided for cluster scheduling. You can select an appropriate YAML example based on the used component, processor type, and job type, and make necessary modifications according to actual requirements before using it.

YAML Description

Compared with a common AscendJob, the MindIE Motor inference job requires the two additional labels: app and jobID. MindIE Server requires NPUs. Ensure that the number of AscendJobs delivered matches the number of prefill instances and decode instances.

• MS Controller and MS Coordinator do not require NPUs and are each deployed as an AscendJob, supporting multiple replicas. The YAML examples for MS Controller and MS Coordinator are as follows.

    apiVersion: mindxdl.gitee.com/v1
    kind: AscendJob
    metadata:
      name: mindie-ms-test-controller
      namespace: mindie
      labels:
        framework: pytorch
        app: mindie-ms-controller   # Indicates the role of MindIE Motor in the Ascend Job task, non-modifiable
        jobID: mindie-ms-test       # Unique identification ID of the current MindIE Motor task in the cluster, configurable by the user based on actual conditions
        ring-controller.atlas: ascend-910b
    spec:
      schedulerName: volcano   # Scheduler selected when Ascend Operator enables "gang" scheduling
      runPolicy:
        schedulingPolicy:      # This field only takes effect when Ascend Operator enables "gang" scheduling and the scheduler is Volcano
          minAvailable: 1      # Total replica count for task running
          queue: default
      successPolicy: AllWorkers
      replicaSpecs:
        Master:
          replicas: 1
          restartPolicy: Always
          template:
            metadata:
              ...

app and jobID are described as follows. For detailed descriptions of other parameters, see YAML Parameter Description.

app: role of MindIE Motor in the AscendJob. Values include mindie-ms-controller, mindie-ms-coordinator, and mindie-ms-server.

jobID: unique ID of the MindIE Motor job in the cluster. You can configure the ID as required.

• YAML example for MindIE Server

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: rings-config-mindie-server-0  # The name must be identical to the name attribute of the following AscendJob. The prefix "rings-config-" is non-modifiable.
      namespace: mindie
      labels:
        jobID: mindie-ms-test
        ring-controller.atlas: ascend-910b
        mx-consumer-cim: "true"
    data:
      hccl.json: |
        {
            "status":"initializing"
        }
  
    apiVersion: mindxdl.gitee.com/v1
    kind: AscendJob
    metadata:
      name: mindie-server-0
      namespace: mindie
      labels:
        framework: pytorch
        app: mindie-ms-server        # Indicates the role of the current MindIE Motor in the Ascend Job task, non-modifiable
        jobID: mindie-ms-test        # The unique identification ID of the current MindIE Motor task in the cluster. Users can configure it based on actual conditions.
        ring-controller.atlas: ascend-910b
      annotations:
        huawei.com/schedule.filter.faultCode: "8C1F8608,4C1F8608,80E01801"       # Add this annotation. For the configuration method, see YAML Parameter Description.
        huawei.com/schedule.filter.faultLevel: "RestartRequest"       # Add this annotation. For the configuration method, see YAML Parameter Description.
    spec:
      schedulerName: volcano   # The scheduler selected when Ascend Operator enables "gang" scheduling
      runPolicy:
        schedulingPolicy:      # This field only takes effect when Ascend Operator enables "gang" scheduling and the scheduler is Volcano
          minAvailable: 2      # Total replica count for task running
          queue: default
      successPolicy: AllWorkers
      replicaSpecs:
        Master:

• For the Atlas 800I A3 SuperPoD Server, the YAML for MindIE Server requires the following modifications:

    apiVersion: mindxdl.gitee.com/v1
    kind: AscendJob
    metadata:
      name: mindie-server-0
      namespace: mindie
      labels:
        framework: pytorch
        app: mindie-ms-server        # Non-modifiable
        jobID: mindie-ms-test        # Unique identification ID of the MindIE Motor job in the cluster. Users can configure it based on actual conditions.
        ring-controller.atlas: ascend-910b
        fault-scheduling: force
      annotations:
        sp-block: "16"         # Add this annotation. For the configuration method, see YAML Parameter Description.
        huawei.com/schedule_policy: "chip2-node16"    # Set the scheduling policy based on the hardware form
        huawei.com/schedule.filter.faultCode: "8C1F8608,4C1F8608,80E01801"       # Add this annotation. For the configuration method, see YAML Parameter Description.
        huawei.com/schedule.filter.faultLevel: "RestartRequest"       # Add this annotation. For the configuration method, see YAML Parameter Description.
    spec:
      schedulerName: volcano   # Volcano selected when Ascend Operator enables "gang" scheduling
      runPolicy:
        schedulingPolicy:      # This field only takes effect when Ascend Operator enables "gang" scheduling and the scheduler is Volcano
          minAvailable: 2      # Total replica count for job running
          queue: default
      successPolicy: AllWorkers
      replicaSpecs:
        Master:
          replicas: 1
          restartPolicy: Always
          template:
            metadata:
              labels:
                ring-controller.atlas: ascend-910b
                app: mindie-ms-server
                jobID: mindie-ms-test
            spec:
              nodeSelector:
                accelerator: huawei-Ascend910

(Optional) Configuring Instance-Level Affinity Scheduling

For the Atlas 800I A3 SuperPoD Server, MindCluster cluster scheduling components support job-level affinity scheduling for MindIE Motor inference jobs. That is, MindIE Server instances can be scheduled to the same physical SuperPoD as much as possible, fully utilizing the HCCS network and accelerating network communication between instances.

For detailed instructions on affinity scheduling rules for logical SuperPoDs, see the UnifiedBus Device Network Description section.

Figure 1 UnifiedBus device network

Configuring Instance-Level Affinity Scheduling

After image preparation is finished, if instance-level affinity scheduling policies are required for MindIE Motor inference jobs during Preparing a Job YAML, perform the following operations simultaneously.

• Specify the sp-block field in the job YAML. The value of sp-block must be consistent with the chip quantity to ensure that the entire job is scheduled to a single physical SuperPoD.

• Ensure that there are reserved nodes in a physical SuperPoD for MindIE Server instance scheduling.

• When sp-fit is set to idlest, MindIE Server instances are scheduled to a more idle physical SuperPoD.

• When podAffinity is set, MindIE Server instances are scheduled to a physical SuperPoD with more affinity pods.

The YAML example is as follows:

apiVersion: mindxdl.gitee.com/v1
kind: AscendJob
metadata:
  name: mindie-server-0
  namespace: mindie
  labels:
    framework: pytorch
    app: mindie-ms-server        # Role of MindIE Motor in AscendJob, non-modifiable
    jobID: mindie-ms-test        # Unique identification ID of the current MindIE Motor job in the cluster, configurable by the user based on actual conditions
    ring-controller.atlas: ascend-910b
    fault-scheduling: force
  annotations:
    sp-block: "16"              # Specifies the sp-block field. The cluster scheduling components will divide logical SuperPoDs based on the partitioning strategy on top of the physical SuperPoD for affinity scheduling of training jobs.
    sp-fit: "idlest"            # SuperPoD scheduling policy. For detailed description, see YAML Parameter Description.
    huawei.com/schedule_policy: "chip2-node16"    # Sets the scheduling policy based on the hardware form.
spec:
  schedulerName: volcano   # Volcano selected when Ascend Operator enables "gang" scheduling
  runPolicy:
    schedulingPolicy:      # This field only takes effect when Ascend Operator enables "gang" scheduling and the scheduler is Volcano
      minAvailable: 2      # Total replica count for job running
      queue: default
  successPolicy: AllWorkers
  replicaSpecs:
    Master:
      restartPolicy: Never
      template:
        metadata:
          labels:
            ring-controller.atlas: ascend-910b
        spec:
          affinity:
            podAffinity:            # Indicates that the logical SuperPoD will be scheduled to the physical SuperPoD with more affinity pods
              preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 100         # Non-modifiable
                podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      jobID: mindie-ms-test  # Label required for affinity pods
                  topologyKey: kubernetes.io/hostname

YAML Parameter Description

The following table describes the YAML parameters in the AscendJob.

Table 1 YAML parameters

Delivering, Viewing, and Deleting an Inference Job

After completing the preparation of the job YAML,perform the following operations:

1. Deliver an inference job.
2. View scheduling results.
3. View the inference job running status.
4. (Optional) Delete a job.

For detailed instructions on the above steps, see the "Cluster Service Deployment > Prefill-Decode Disaggregation Deployment > Installation and Deployment > Deploying a Single-Server Prefill-Decode Disaggregation Service Using kubectl" section in the MindIE Motor Development Guide.

global-ranktable Description

ClusterD listens to the information of the MS Controller and MS Coordinator job pods and the changes of the ConfigMap corresponding to each hccl.json file, and generates global-ranktable in real time. Some fields in global-ranktable are consistent with those in the hccl.json file. For details about hccl.json, see hccl.json File Description.

• global-ranktable example of the Atlas A2 training series products

  {
      "version": "1.0",
      "status": "completed",
      "server_group_list": [
          {
              "group_id": "2",
              "deploy_server": "0",
              "server_count": "1",
              "server_list": [
                  {
                      "device": [
                          {
                              "device_id": "x",
                              "device_ip": "xx.xx.xx.xx",
                              "device_logical_id": "x",
                              "rank_id": "x"
                          }
                      ],
                      "server_id": "xx.xx.xx.xx",
                      "server_ip": "xx.xx.xx.xx"
                  }
              ]
          }
      ]
  }

• global-ranktable example of the Atlas A3 training series products

  {
      "version": "1.2",
      "status": "completed",
      "server_group_list": [
          {
              "group_id": "2",
              "deploy_server": "1",
              "server_count": "1",
              "server_list": [
                  {
                      "device": [
                          {
                              "device_id": "0",
                              "device_ip": "xx.xx.xx.xx",
                              "super_device_id": "xxxxx",
                              "device_logical_id": "0",
                              "rank_id": "0"
                          }
                      ],
                      "server_id": "xx.xx.xx.xx",
                      "server_ip": "xx.xx.xx.xx"
                  }
              ],
              "super_pod_list": [
                  {
                      "super_pod_id": "0",
                      "server_list": [
                          {
                              "server_id": "xx.xx.xx.xx"
                          }
                      ]
                  }
              ]
          }
      ]
  }

Table 1 global-ranktable field description