Deploying a Multi-Node Prefill-Decode Disaggregation Service Using kubectl

kubectl create namespace mindie

For prefill-decode disaggregation deployment, set deploy_mode to pd_separate.

"deploy_mode": "pd_separate"

If heterogeneous deployment is required, set is_heterogeneous to true.

"is_heterogeneous": true

• The request_coordinator_tls_enable, request_server_tls_enable, http_server_tls_enable, and cluster_tls_enable parameters in the ms_controller.json file specify whether to enable HTTPS. You are advised to enable HTTPS (by setting these parameters to true) to ensure communication security. If HTTPS is disabled, high network security risks exist.
  • true: HTTPS enabled for MindIE components in the cluster. You need to import certificates to the container and configure the corresponding certificate paths.
  • false: HTTP enabled for MindIE components in the cluster. You do not need to prepare certificates.
• default_p_rate and default_d_rate in the ms_controller.json file control the ratio of prefill nodes to decode nodes in the cluster. The default values are both 0. The optimal ratio is automatically determined based on the model, hardware, and service information. You can also set the ratio to the actual number of prefill nodes and decode nodes based on the scenario situation. If the environment variables MINDIE_MS_P_RATE and MINDIE_MS_D_RATE are set, their values are preferentially read. For details, see Environment Variable.

For prefill-decode disaggregation deployment, set deploy_mode to pd_separate.

"deploy_mode": "pd_separate"

The controller_server_tls_enable, request_server_tls_enable, mindie_client_tls_enable, and mindie_mangment_tls_enable parameters in the ms_coordinator.json file specify whether to enable HTTPS. You are advised to enable HTTPS (by setting these parameters to true) to ensure communication security. If HTTPS is disabled, high network security risks exist.

• true: HTTPS enabled for MindIE components in the cluster. You need to import certificates to the container and configure the corresponding certificate paths.
• false: HTTP enabled for MindIE components in the cluster. You do not need to prepare certificates.

tls_enable specifies whether to enable HTTPS. If the MindIE components in the cluster use the HTTPS interface, set tls_enable to true, import certificates to the container, and configure the corresponding certificate paths. To use the HTTP interface, set tls_enable to false. No certificate file is required.

You are advised to enable tls_enable to ensure communication security. If tls_enable is disabled, high network security risks exist.

Find the mindie_server.yaml, mindie_ms_coordinator.yaml, mindie_ms_controller.yaml, and mindie_server_heterogeneous.yaml (configured only in heterogeneous scenarios) files in the deployment directory among the deployment script directories.

• The script is for reference only. You need to ensure the security of the pod container. In the actual production environment, harden the security of the image and pod.
• When using kubectl to deploy a Deployment, you can modify the YAML configuration file of the Deployment. Do not use dangerous configurations, and ensure that a secure image (non-root user) is used to configure secure pod contexts.
• You must mount secure paths (non-soft links, non-dangerous system paths, and non-service sensitive paths) and set proper directory permissions to prevent public directories such as /home from being mounted and prevent container escape caused by tampering by unauthorized users.

• Fields to be configured in the mindie_server.yaml file:
  • replicas:
    • Homogeneous scenario: total number of prefill and decode instances. If there are three prefill instances and one decode instance, set this parameter to 4.
    • Heterogeneous scenario: number of prefill instances. If there are three prefill instances, set this parameter to 3.
  • huawei.com/Ascend910: number of NPUs occupied by a prefill/decode instance. The value must be the same as that of worldSize in the config.json file of MindIE LLM.
  • image: image name.
    • MindIE and ATB Models used together: see MindIE and ATB Models Used Together.
    • MindIE and MindSpore used together: see MindIE and MindSpore Used Together.
  • startupProbe: startup probe. The default startup time is 500 seconds. If the service fails to be started within the time, the pod automatically restarts. Set a proper startup time as required.
  • affinity: anti-affinity deployment configuration. By default, it is disabled. After the comment is deleted, it is enabled. After this function is enabled, each Server pod is deployed on different compute nodes in anti-affinity mode. Multiple pods are not deployed on the same node.
  • nodeSelector: selects the node to be scheduled through node parameters.
    • hardware_type: required only for heterogeneous deployment, which is used to enable Controller to determine the heterogeneous inference role.
    • hardware-type: required only for heterogeneous deployment. It is the node label managed by the Kubernetes cluster and is used to allocate heterogeneous device resources. Before heterogeneous deployment, run the following command to label heterogeneous compute nodes:

      kubectl label node xx_node hardware-type=xx_device

      

      In heterogeneous scenarios, the parameter values must meet the following requirements:

      • hardware_type: 800I A2(32G) for a prefill node, and 800I A2(64G) for a decode node.
      • hardware-type: The value of this parameter cannot contain spaces.
  • MINDIE_LOG_TO_FILE: whether to write logs of each MindIE component to a file.

    The default value is 1, indicating that logs are written into a file. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_TO_STDOUT: whether to print logs of each MindIE component.

    The default value is 1, indicating that logs are printed. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_LEVEL: log level of each MindIE component.

    The default value is INFO. The log levels are [CRITICAL, ERROR, WARN, INFO, DEBUG].

• Pay special attention to the image name and log settings in the mindie_ms_coordinator.yaml and mindie_ms_controller.yaml files.
  • MindIE and ATB Models used together: see MindIE and ATB Models Used Together.
  • MindIE and MindSpore used together: see MindIE and MindSpore Used Together.
  • MINDIE_LOG_TO_FILE: whether to write logs of each MindIE component to a file.

    The default value is 1, indicating that logs are written into a file. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_TO_STDOUT: whether to print logs of each MindIE component.

    The default value is 1, indicating that logs are printed. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_LEVEL: log level of each MindIE component.

    The default value is INFO. The log levels are [CRITICAL, ERROR, WARN, INFO, DEBUG].

  

  • Pay special attention to the livenessProbe parameter in the mindie_ms_coordinator.yaml file. For high-concurrency inference requests, the probe may time out and Kubernetes may identify that the pod is not alive. As a result, Kubernetes restarts the Coordinator container. Exercise caution when enabling livenessProbe.
  • To enable the fault recovery function of MindIE Controller, the directory specified by name: status-data under the volumes parameter in the mindie_ms_controller.yaml file must exist on the compute node to be deployed (specified by nodeSelector). The mount path of name: status-data under volumeMounts must be /MindIE Motor installation path/logs.
• Fields to be configured in the mindie_server_heterogeneous.yaml file:
  • replicas: number of decode instances. If there are four decode instances, set this parameter to 4.
  • huawei.com/Ascend910: number of NPUs occupied by a prefill/decode instance. If there are two NPUs, set this parameter to 2.
  • image: image name.
    • MindIE and ATB Models used together: see MindIE and ATB Models Used Together.
    • MindIE and MindSpore used together: see MindIE and MindSpore Used Together.
  • startupProbe: startup probe. The default startup time is 500 seconds. If the service fails to be started within the time, the pod automatically restarts. Set a proper startup time as required.
  • nodeSelector: selects the node to be scheduled through node parameters.
    • hardware_type: required only for heterogeneous deployment, which is used to enable Controller to determine the heterogeneous inference role. The value of this parameter must be different from that of hardware_type in the mindie_server.yaml file.
    • hardware-type: required only for heterogeneous deployment. It is the node label managed by the Kubernetes cluster and is used to allocate heterogeneous device resources. Before heterogeneous deployment, run the following command to label heterogeneous compute nodes. The value of this parameter must be different from that of hardware-type in the mindie_server.yaml file.

      kubectl label node xx_node hardware-type=xx_device2

      

      In heterogeneous scenarios, the parameter values must meet the following requirements:

      • hardware_type: 800I A2(32G) for a prefill node, and 800I A2(64G) for a decode node.
      • hardware-type: The value of this parameter cannot contain spaces.
  • MINDIE_LOG_TO_FILE: whether to write logs of each MindIE component to a file.

    The default value is 1, indicating that logs are written into a file. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_TO_STDOUT: whether to print logs of each MindIE component.

    The default value is 1, indicating that logs are printed. The value range is [false, true] or [0, 1].

  • MINDIE_LOG_LEVEL: log level of each MindIE component.

    The default value is INFO. The log levels are [CRITICAL, ERROR, WARN, INFO, DEBUG].

Configure the MindIE installation directory in the container. Change the value of MINDIE_USER_HOME_PATH based on the actual installation path during image creation. For example, if the installation path is /xxx/Ascend/mindie, set the value to /xxx.

export MINDIE_USER_HOME_PATH={Image installation path}

Before starting the service, you are advised to use the pre-check tool of MindStudio to verify the fields in the configuration file and check the validity of the configuration. For details, see Link.

Run the following command to start the cluster:

bash deploy.sh

After the command is executed, wait until global_ranktable.json is generated. If the generation is blocked for a long time, press Ctrl+C to interrupt the process, and check the pod status of the cluster for debugging.

The following is an example of global_ranktable.json. For details about the parameters, see Table 4.

{
  "version": "1.0",
  "server_group_list": [
    {
      "group_id": "2",
      "server_count": "2",
      "server_list": [
        {
          "server_id": "xxx.xxx.xxx.1",
          "server_ip": "xxx.xxx.xxx.1",
          "device": [
            {
              "device_id": "0",
              "device_ip": "xxx.xxx.xxx.1",
              "device_logical_id": "0"
            }
          ],
          "hardware_type": "800I A2(32G)"
        },
        {
          "server_id": "xxx.xxx.xxx.2",
          "server_ip": "xxx.xxx.xxx.2",
          "device": [
            {
              "device_id": "1",
              "device_ip": "xxx.xxx.xxx.2",
              "device_logical_id": "1"
            }
          ],
          "hardware_type": "800I A2(64G)"
        }
      ]
    },
    {
      "group_id": "1",
      "server_count": "1",
      "server_list": [
        {
          "server_id": "xxx.xxx.xxx.1",
          "server_ip": "xxx.xxx.xxx.1"
        }
      ]
    },
    {
      "group_id": "0",
      "server_count": "1",
      "server_list": [
        {
          "server_id": "xxx.xxx.xxx.1",
          "server_ip": "xxx.xxx.xxx.1"
        }
      ]
    }
  ],
  "status": "completed"
}

• If the deployment fails, delete the cluster by referring to 11 and deploy it again.
• By default, the cluster updates the ConfigMap mounted to the container every 60 seconds. If it takes long to print the message "status of ranktable is not completed" in the container, you can change the interval for synchronizing the ConfigMap by kubelet on each compute node to be scheduled. That is, modify the syncFrequency parameter in the /var/lib/kubelet/config.yaml file to reduce the period to 5 seconds. Note that this modification may affect the cluster performance.

  syncFrequency: 5s

  Restart kubelet:

  swapoff -a
  systemctl restart  kubelet.service
  systemctl status kubelet

• Ensure that Docker is configured with the maximum specifications for writing standard output streams to files to prevent pods from being evicted due to full drive space.

  After modifying the Docker configuration file on the compute node where the service is to be deployed, restart Docker.

  1. Open the daemon.json file.

     vim /etc/docker/daemon.json

     Add log-opts to the daemon.json file as follows.

     "log-opts":{"max-size":"500m", "max-file":"3"}

     Parameters:

     max-size=500m indicates that the maximum size of a container log file is 500 MB.

     max-file=3 indicates that a container has a maximum of three log files. If the number of log files exceeds three, the log files are automatically rotated.

  2. Restart Docker.

     systemctl daemon-reload
     systemctl restart docker

To enable heterogeneous cluster inference, run the following command:

bash deploy.sh heter

kubectl get pods -n mindie

For example, if four Server instances are started, the following information is displayed:

1
2
3
4
5
6
7

NAME                                     READY   STATUS    RESTARTS   AGE    IP               NODE       NOMINATED NODE   READINESS GATES
mindie-ms-controller-7845dcd697-h4gw7    1/1     Running   0          145m   xx.xx.xx.xx    ubuntu10   <none>           <none>
mindie-ms-coordinator-6bff995ff8-l6fwz   1/1     Running   0          145m   xx.xx.xx.xx    ubuntu10   <none>           <none>
mindie-server-7b795f8df9-2xvh4           1/1     Running   0          145m   xx.xx.xx.xx   ubuntu     <none>           <none>
mindie-server-7b795f8df9-j4z7d           1/1     Running   0          145m   xx.xx.xx.xx   ubuntu     <none>           <none>
mindie-server-7b795f8df9-v2tcz           1/1     Running   0          145m   xx.xx.xx.xx   ubuntu     <none>           <none>
mindie-server-7b795f8df9-vl9hv           1/1     Running   0          145m   xx.xx.xx.xx   ubuntu     <none>           <none>

• mindie-ms-controller: Controller
• mindie-ms-coordinator: Coordinator
• mindie-server: Server

If the pod is in the Running status, the pod container has been successfully scheduled to a node and started. However, you need to further check whether the service program is started successfully.

• You can use the provided log.sh script to query the standard output logs of pods and check whether a program exception occurs.

  bash log.sh

  

  If heterogeneous cluster inference is configured, run the following command:

  bash log.sh heter

• To query the logs of a specific pod (for example, mindie-server-7b795f8df9-vl9hv), run the following command:

  kubectl logs mindie-server-7b795f8df9-vl9hv -n mindie

• To access the container to search for more information, run the following command:

  kubectl exec -it mindie-server-7b795f8df9-vl9hv -n mindie -- bash

• To confirm the pods corresponding to prefill and decode nodes, run the following command after Controller is started (for example, mindie-ms-controller-7845dcd697-h4gw7 is in the READY 1/1 state):

  kubectl logs mindie-ms-controller-7845dcd697-h4gw7 -n mindie | grep UpdateServerInfo

  After the pod IP addresses of the prefill and decode nodes are queried, you can find the corresponding pods based on the IP addresses in the command output of the preceding command for querying the pod status.