Installation Preparation
To reduce manual operations, you do not need to create a new user on the host. You only need to ensure that no other user occupies UID 9000. For components whose startup user is hwMindX (see Table 1), decide how to create the user based on the actual situation:
Create a user: (1) You can create the
hwMindXuser using the following command; (2) You can also create a custom user. In this case, replace the username and UID in the following command with the actual values, and modify the UID in theuseraddcommand in the Dockerfile of the corresponding component to match the actual value.Using an existing user in the environment: You need to run
usermod -a -G HwHiAiUser {your_username}, and modify the Dockerfile of the corresponding component to replace the UID in theuseraddcommand in the Dockerfile with the UID of the existing user.
- Creating users on other operating systems:
HwHiAiUseris the software runtime user required by the driver or CANN package.- Run the
getent passwdcommand to check whether the UID and GID ofHwHiAiUserare consistent across all physical machines (storage nodes, management nodes, and compute nodes) and inside containers, and that both are 1000. If they are occupied, the service may become unavailable. For details, see User UID or GID Occupied. - If the user is not created on the host, the Dockerfile will create a user with UID 9000 inside the container by default. Processes inside the container will run with this UID and map to the host. Ensure that this UID is not occupied by another user on the host machine.
Table 1 Component user description
Create the parent log directory for components and the log directories for each component on the corresponding nodes, and set the owner and permissions for the directories.
Run the following command to create the parent log directory for components on each node according to Table 1 Cluster scheduling component log path list.
mkdir -m 755 /var/log/mindx-dl chown root:root /var/log/mindx-dlCreate the corresponding log directories based on the specific components used. The username must be the user created in the Creating Users step. The following example uses the
hwMindXuser; replacehwMindXin the following commands with the actual username you are using. If no user has been created on the host, use the numeric UID instead of the username in thechowncommand, for example,chown 9000:9000 /var/log/mindx-dl/clusterd. When modifying the startup user for an installed component, usechown -Rto recursively modify the owner of the log directory; otherwise, existing log files in the directory will still belong to the original user, which may prevent the new user from reading them.Table 1 Cluster scheduling component log path list
In a K8s cluster, if a node containing an Ascend AI Processor is used as the K8s management node, this node serves as both a management node and a compute node. In addition to the labels required for the management node, you also need to apply the relevant labels for the compute node based on the type of Ascend AI Processor on the node. In a production environment, the management node is typically a general-purpose server and does not contain an Ascend AI Processor.
Run the following command on any node to query the node name.
kubectl get nodeSample output:
NAME STATUS ROLES AGE VERSION ubuntu Ready worker 23h v1.17.3Label the corresponding nodes based on the label information in Table 1 to facilitate the cluster scheduling components in scheduling across various types of worker nodes.
kubectl label nodes Hostname LabelTaking the hostname
ubuntuand the labelmasterselector=dls-master-nodeas an example, the command is as follows.kubectl label nodes ubuntu masterselector=dls-master-nodeThe sample output is as follows, indicating a successful operation.
node/ubuntu labeledNOTE
- For a detailed description of each node label in Table 1, see the K8s Native Object Description section.
- Configure all the labels listed in Table 1 based on the node type and product type.
- {xxx} uses
910as an example chip model.
Table 1 Label information corresponding to nodes
The NodeD, Resilience Controller, ClusterD, Infer Operator, and Ascend Operator components for cluster scheduling run in the
mindx-dlnamespace of K8s. Run the following command on the management node of K8s to create the corresponding namespace.kubectl create ns mindx-dlTo report SuperPoD information, pingmesh configuration, and public fault information, MindCluster requires the manual creation of a namespace named
cluster-system. Run the following command on the management node of K8s.kubectl create ns cluster-systemThe namespace for NPU Exporter is
npu-exporter; the namespace for Volcano isvolcano-system; the namespace for Ascend Device Plugin iskube-system. The namespaces for the preceding components are created by the system, and you do not need to create them again.
You can prepare images in the following two ways. After obtaining the images, create node labels, users, log directories, and namespaces for the corresponding components to be installed in sequence.
(Recommended) Create Images: Taking Ascend Operator as an example, this section describes the steps for creating images required for container deployment of cluster scheduling components. The Dockerfile in the software package is for reference only, and you can create customized images based on this example.
Pull Images from Ascend Image Repository: You can obtain pre-built images of cluster scheduling components from the image repository.
NOTE
- After pulling or creating images, perform security hardening in a timely manner, such as fixing vulnerabilities in the base image and those introduced by third-party dependencies.
- Import the images into the container runtime used by K8s. For example, K8s versions 1.24 and later use Containerd as the default container runtime. After pulling or creating images, you need to import them into Containerd.
- The running user for NPU Exporter and Ascend Device Plugin is
root. TheLD_LIBRARY_PATHenvironment variable is configured in their corresponding Dockerfiles, and its value includes paths related to driver libraries. The components will use files in these paths at runtime. It is recommended that the running user specified during driver installation berootto avoid privilege escalation risks caused by user inconsistency. - For components whose startup user is
hwMindX, if a custom user or an existing user is used during user creation, you need to modify the UID in theuseraddcommand in the corresponding component's Dockerfile before image creation to make it consistent with the user UID on the host.
In the Obtaining Software Packages chapter, obtain the software packages of the cluster scheduling components that need to be installed.
After decompressing the software packages, upload them to any directory on the image creation server. Taking Ascend Operator as an example, place them in the
/home/ascend-operatordirectory. The directory structure is as follows.root@node:/home/ascend-operator# ll total 41388 drwxr-xr-x 2 root root 4096 Aug 26 20:20 ./ drwxr-xr-x 6 root root 4096 Aug 26 20:20 ../ -r-------- 1 root root 41992192 Aug 26 02:02 agreement.txt -r-x------ 1 root root 41992192 Aug 26 02:02 ascend-operator* -r-------- 1 root root 372291 Aug 26 02:02 ascend-operator-v{version}.yaml -r-------- 1 root root 482 Aug 26 02:02 Dockerfile -r-------- 1 root root 1024 Aug 26 02:02 Dockerfile.openeulerNOTE
If NPU Exporter and Ascend Device Plugin are deployed as images on the Atlas 200I SoC A1 core board, perform the following operations.
- When creating the image, check the UID and GID of the
HwHiAiUser,HwDmUser, andHwBaseUserusers on the host, and record the values of these GIDs and UIDs. - Check whether the GID and UID specified when creating the
HwHiAiUser,HwDmUser, andHwBaseUserusers inDockerfile-310P-1usocare consistent with those on the host. If they are consistent, no modification is required; if they are inconsistent, manually modify theDockerfile-310P-1usocfile to make them consistent. At the same time, ensure that the GID and UID values of theHwHiAiUser,HwDmUser, andHwBaseUserusers are consistent on each host.
- When creating the image, check the UID and GID of the
Check whether the following base images exist on the node used for creating cluster scheduling component images.
If you need to use the Ubuntu base image for building, check whether the Ubuntu image exists. Run the
docker images | grep ubuntucommand to check the Ubuntu image. The image sizes differ between ARM architecture and x86_64 architecture.ubuntu 22.04 6526a1858e5d 2 years ago 64.2MBIf you need to use the openEuler image as the base image for building, check whether the openEuler image exists. Run the
docker images | grep openeulercommand to check the openEuler image. The image sizes differ between ARM and x86_64 architectures.openeuler/openeuler 24.03-lts 2fc1d956e7ed 2 years ago 205MBIf you need to install Volcano, check whether the alpine image exists. Run the
docker images | grep alpinecommand to check. The sample output is as follows. The image sizes differ between ARM and x86_64 architectures.alpine latest a24bb4013296 2 years ago 5.57MB
If the preceding base images do not exist, use the relevant commands in Table 1 to pull the base images (pulling images requires the server to have internet access).
Table 1 Commands for obtaining base images
Base Image
Image Pull Command
Description
ubuntu:22.04
docker pull ubuntu:22.04
The system architecture is automatically identified during the pull.
openeuler/openeuler:24.03-lts
docker pull openeuler/openeuler:24.03-lts
The system architecture is automatically identified during the pull.
alpine:latest
- x86_64 architecture
docker pull alpine:latest
- ARM architecture
docker pull arm64v8/alpine:latest docker tag arm64v8/alpine:latest alpine:latest
-
Go to the component extraction directory and run the
docker buildcommand to create the image. For command details, see Table 2.Table 2 Image creation commands for each component
Taking the image creation of Ascend Operator as an example, run the docker build --no-cache -t ascend-operator:v{version} . command to create the image. The sample output is as follows. Note: Do not omit the "." at the end of the command.
```ColdFusion
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/
Sending build context to Docker daemon 42.37MB
Step 1/5 : FROM ubuntu:22.04 as build
---> 1f37bb13f08a
Step 2/5 : RUN useradd -d /home/hwMindX -u 9000 -m -s /usr/sbin/nologin hwMindX && usermod root -s /usr/sbin/nologin
---> Running in d43f1927b1fd
Removing intermediate container d43f1927b1fd
---> 9f1d64e06ee6
Step 3/5 : COPY ./ascend-operator /usr/local/bin/
---> 5022b58c516e
Step 4/5 : RUN chown -R hwMindX:hwMindX /usr/local/bin/ascend-operator && chmod 500 /usr/local/bin/ascend-operator && chmod 750 /home/hwMindX && echo 'umask 027' >> /etc/profile && echo 'source /etc/profile' >> /home/hwMindX/.bashrc
---> Running in a781bde3dc56
Removing intermediate container a781bde3dc56
---> 3d7e2ee7a3bd
Step 5/5 : USER hwMindX
---> Running in 338954be8d99
Removing intermediate container 338954be8d99
---> 103f6a2b43a5
Successfully built 103f6a2b43a5
Successfully tagged ascend-operator:v{version}
```You can skip this step in the following scenarios.
- The created cluster scheduling component images have been uploaded to a private image repository, and each node can pull the cluster scheduling component images from the private image repository.
- The corresponding component images have been created on each node where the cluster scheduling components are installed.
In other scenarios, you need to manually distribute each component image to each node. Taking NodeD as an example, use the offline image package to distribute the image to other nodes.
Save the created image as an offline image.
docker save noded:v{version} > noded-v{version}-linux-aarch64.tarCopy the image to other nodes.
scp noded-v{version}-linux-aarch64.tar root@{Target node IP}:Save pathLog in to each node as the
rootuser and load the offline image.docker load < noded-v{version}-linux-aarch64.tar
(Optional) Import the offline image into Containerd. This step applies to scenarios where the container runtime is Containerd. It can be skipped in other scenarios.
Taking NodeD as an example, execute the following command.
ctr -n k8s.io images import noded-v{version}-linux-aarch64.tar
Pull Images from the Ascend Image Repository
After ensuring the server can access the internet, visit the Ascend Image Repository.
In the left navigation pane, select the task type as "Cluster Scheduling", and then select the corresponding component images according to the following table. The pulled images need to be renamed before they can be deployed using the component startup YAML. For details, see Step 3.
Table 3 Image list
If you do not have download permissions, apply for permissions as prompted on the page. After submitting the application, wait for the administrator's approval. Once approved, you can download the image.
The cluster scheduling images pulled from the Ascend image repository have different names from those in the component startup YAML files. You need to rename the pulled images before starting the components. Follow the steps below to rename the images obtained in Step 2. It is also recommended to delete the images with the original names. The specific operations are as follows:
Rename images (users need to select the corresponding command based on the component used).
docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v26.0.0 ascend-operator:v26.0.0 docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v26.0.0 npu-exporter:v26.0.0 docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v26.0.0 ascend-k8sdeviceplugin:v26.0.0 # When using Volcano version 1.9.0, you need to change the image tag to v1.9.0-v26.0.0. Starting from MindCluster v26.1.0, the image tag must be changed to v1.7.0-v26.1.0. docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v26.0.0 volcanosh/vc-controller-manager:v1.7.0 docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v26.0.0 volcanosh/vc-scheduler:v1.7.0 docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v26.0.0 noded:v26.0.0 docker tag swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v26.0.0 clusterd:v26.0.0(Optional) Delete images with the original names (users need to select the corresponding command based on the component used).
docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-operator:v26.0.0 docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/npu-exporter:v26.0.0 docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/ascend-k8sdeviceplugin:v26.0.0 # When using Volcano v1.9.0, change the image tag to v1.9.0-v26.0.0. docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-controller-manager:v1.7.0-v26.0.0 docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/vc-scheduler:v1.7.0-v26.0.0 docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/noded:v26.0.0 docker rmi swr.cn-south-1.myhuaweicloud.com/ascendhub/clusterd:v26.0.0
(Optional) Import the offline image into Containerd. This step applies to scenarios where the container runtime is Containerd. Skip this step for other scenarios.
Taking the NodeD component as an example, use the offline image package and perform the following steps.
Save the created image as an offline image.
docker save noded:v{version} > noded-v{version}-linux-aarch64.tarImport the offline image into Containerd.
ctr -n k8s.io images import noded-v{version}-linux-aarch64.tar