Preparing for MindCluster Cluster Scheduling Component Upgrade

Restrictions

• When using MindCluster Ascend Deployer to upgrade MindCluster cluster scheduling components, use MindCluster Ascend Deployer to upgrade other software dependencies.
• When using MindCluster Ascend Deployer to upgrade MindCluster cluster scheduling components, you are advised to select a worker node as the executor.
• When upgrading MindCluster, ensure that the available disk space of the Docker container, file system, or root directory in the system is greater than 30% after adding an estimated 18 GB (for the MindCluster image and training and inference images) to the used space.
• Creating a cluster between nodes running different OSs may cause the MindCluster upgrade to fail.

Prerequisites

• You have performed the operations in Configuring the Parameter Plane Network. Before cluster training, configure the hccn_tool network for the training nodes. Refer to 1 to modify only the HCCN variable configuration area.
• Ensure that DNS has been configured for all servers in the cluster. It is recommended that the DNSs of all target devices be the same. If DNS is not configured, configure it by referring to Configuring DNS.

Configuring inventory_file

1. Log in to the MindCluster Ascend Deployer executor.
2. Configure the IP addresses and usernames of target devices on the MindCluster Ascend Deployer executor.
   Go to the ascend-deployer/ascend_deployer directory, open inventory_file to add the configuration, and run the :wq command to save the file and exit.

   1. Set variables for the master, worker, other_build_image (optional), and npu_node (optional) based on Table 1.
      

      • If the architecture of a server in a Kubernetes cluster is different from that of the master node and MindCluster cluster scheduling components are deployed on the node (or multiple heterogeneous nodes), you need to configure other_build_image in inventory_file. For details, see Configuring other_build_image.
      • When configuring MindCluster cluster scheduling for a Kubernetes cluster, you need to pay attention to the different NPU hardware forms in the cluster. You can set npu_node to specify the feature server information of different hardware. If npu_node is not configured, the NPU hardware type of the first node configured under worker in inventory_file is used by default. npu_node must belong to the worker group, and providing only the IP address is sufficient. Parameters in the worker group will overwrite the npu_node configuration based on the IP address.
      • If a server serves as both the master node and worker node, you need to enter the server information under both master and worker. Enter the other servers into their respective groups based on their designated node roles.

      Table 1 Parameter description

      Parameter	Required or Not	Description
      IP	Yes	Server IP address. You can enter multiple IP addresses. The details are described as follows: index: sequence number of the server within the IP address segment, for example, 1.1.1.1-1.1.1.3 set_hostname="master-{index}", where 1.1.1.1 corresponds to set_hostname=master-1. The value is of the Int type. The default value is 1. step_len: IP address step. The last IP address is retained even if it is not within the step. For example, if 1.1.1.1-1.1.1.6 step_len=2 is set, 1.1.1.1,1.1.1.3,1.1.1.5,1.1.1.6 is output. Parameters can be parsed in the batch input configuration {}, including data calculation and conversion. After the parsing, the output is a character string. For example, if 1.1.1.1-1.1.1.3 set_hostname="master-{str(index+20)+'x'}" is entered, the parsed host information is 1.1.1.1 set_hostname="master-21x" .... The IP address segment parameters support only Python 3. When IPv6 is used, the IP address cannot be passed as a dynamic parameter.
      ansible_ssh_user	Yes	Account for logging in to a remote server using SSH. The account must be root.
      ansible_ssh_pass	No	Password for logging in to a remote server using SSH. If SSH key-based authentication is configured and the root user is allowed for login, you do not need to set this parameter.
      ansible_ssh_port	No	Port for SSH connection. You do not need to set this parameter when the default port number 22 is used. If a non-default port is used, you need to configure this parameter.
      set_hostname	No	Name of a node in a Kubernetes cluster. You are advised to use the master-1 or worker-1 format to fill in the names in sequence. If a Kubernetes cluster exists, the parameter value must be the name of the node in the cluster. The name must be in lowercase and cannot be entered arbitrarily. If the value of set_hostname is the same as the name of the Kubernetes node, you do not need to set this parameter. If they are different, you need to set this parameter.
      Description Configure the host of the master node as the default Kubernetes controller node. The number of master nodes must be an odd number.

   2. Configure global variables under the [all:vars] field.

      Table 2 Parameter description

      Parameter	Required or Not	Description
      SCALE	No	Whether to perform batch deployment. The default value is false. If the Kubernetes cluster scale is large, you can deploy the components in batches. Set this parameter to true in the second and subsequent batches. During batch deployment, the versions of the MindCluster cluster scheduling components in the entire cluster must be the same.
      RUNNER_IP	No	IP address of the MindCluster Ascend Deployer executor. Ensure that the IP address can communicate with other target devices. You are advised to set this parameter when the executor contains multiple network segments and local IP addresses.

      Example:

      [master]
      xx.xxx.xx.x1 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx" set_hostname="master-1"     # Replace it with the IP address of the target device.
      
      [worker]
      xx.xxx.xx.x1-xx.xxx.xx.x3 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx"
      xx.xxx.xx.x5-xx.xxx.xx.x9 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx" set_hostname="worker-{ip}-{index}" step_len=2
      
      [npu_node]
      xx.xxx.xx.x3 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx" set_hostname="worker-1"      # npu_node is used to set the feature server of different hardware.
      xx.xxx.xx.x4 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx" set_hostname="worker-2"