[object Object]

This section uses two Atlas 800T A2 training servers (one as the management node and one as the compute node) as an example to guide developers through quickly installing NodeD, Ascend Device Plugin, Ascend Docker Runtime, Volcano, ClusterD, and Ascend Operator, and using the full-NPU scheduling feature to quickly submit a training job.

[object Object]

Table 1 Key procedures

[object Object]undefined[object Object]

Ensure that the cluster environment has been set up.

• Kubernetes has been installed on all nodes, with supported versions 1.17.x~1.34.x. If you need to install Volcano, install Kubernetes version 1.19.x or later. For specific Kubernetes versions, see . To obtain the software package, see the .

• Docker has been installed on all nodes, with supported versions 18.09.x~28.5.1. To obtain the software package, see the .

• The corresponding firmware and drivers have been installed on all nodes.

• Check whether npu-smi and hccn_tool tools can run normally on the host.

  [object Object]

[object Object]

The following uses an Atlas 800T A2 training server as an example. For detailed installation steps and parameter descriptions for all components, see .

1. Log in to the compute node or management node as the [object Object] user and create the component installation directories.

   1. Run the following commands in sequence to create the installation directories on the compute node. The following directories are examples only.

      [object Object]

   2. Run the following commands in sequence to create the installation directories on the management node. The following directories are examples only.

      [object Object]

2. Download software packages with your desired architecture. The AArch64 architecture is used as an example.

   1. Run the following commands in sequence to obtain the NodeD, Ascend Device Plugin, and Ascend Docker Runtime installation packages on the compute node and decompress them.

      [object Object]

   2. Run the following commands in sequence on the management node to obtain the Volcano, ClusterD, and Ascend Operator installation packages.

      [object Object]

3. Build component images.

   1. Run the following commands in sequence to pull the base images on the compute node.

      [object Object]

   2. Run the following commands in sequence to pull base images on the management node.

      [object Object]

   3. Run the following commands in sequence to build component images on the compute node.

      [object Object]

   4. Run the following commands in sequence to build component images on the management node.

      [object Object]

4. Create node labels.

   [object Object]

   1. Run the following command on the Kubernetes management node to query the node name.

      [object Object]

      Example output:

      [object Object]

   2. Run the following commands in sequence to create a node label for the compute node ([object Object] as an example).

      [object Object]

   3. Run the following command to create a node label for the management node ([object Object] as an example).

      [object Object]

5. Create a user.

   [object Object]

   1. Run the following commands in sequence to create a username on the compute node.

      [object Object]

   2. Run the following command to create a username on the management node.

      [object Object]

6. Create log directories. Custom log directories are not supported.

   [object Object]

   1. Run the following commands in sequence to create log directories on the compute node.

      [object Object]

   2. Run the following commands in sequence to create log directories on the management node.

      [object Object]

7. Run the following command on any node to create the namespace.

   [object Object]

8. Install components.

   1. Run the following commands in sequence to install Ascend Docker Runtime on the host of the compute node.

      [object Object]

   2. Run the following commands in sequence to copy the component startup YAML files of the compute node to the installation directory of the corresponding component on the management node:

      [object Object]

   3. On the management node, run the following commands in sequence to install components.

      [object Object]

      Take NodeD as an example. If the following information is displayed, the component is successfully installed.

      [object Object]

   4. Run the following command on the management node to check whether the component is started:

      [object Object]

      Take NodeD as an example. If Running is displayed in the command output, the component is started successfully.

      [object Object]

[object Object]

1. Prepare an image.

   Download the ascend-pytorch training image (24.0.X) from the according to the system architecture (Arm/x86_64). Modify the training base image by changing the default user in the container to [object Object]. The image does not contain training scripts, code, or other files. During training, files such as training scripts and code are typically mapped into the container using the mount method.

2. Perform script adaptation.

   1. [object Object][object Object]Download "ResNet50_ID4149_for_PyTorch" from the master branch of the as the training code.

   2. Prepare the dataset corresponding to ResNet-50 on your own, and comply with the corresponding specifications when using it.

   3. The administrator uploads the dataset to the storage node. Go to the [object Object] directory and upload the dataset to any location, such as [object Object].

      [object Object]

   4. Decompress the training code downloaded in to the local machine, and upload the [object Object] directory from the decompressed training code to the environment, for example, to the [object Object] path.

   5. In the [object Object] path, comment out the following code in [object Object].

      [object Object]

   6. Go to the repository, switch to the corresponding version branch according to mindcluster-deploy Open-Source Repository Version Description, obtain the [object Object] file from the [object Object] directory, and construct the following directory structure under the [object Object] path.

      [object Object]

3. Prepare the job YAML.

   1. Go to the repository. Based on the , switch to the corresponding version branch and obtain the [object Object] file from the [object Object] directory ({xxx} indicates the chip model). The example defaults to a single-server single-device job.

   2. Modify the example YAML and upload it to any file path after modification. For detailed descriptions of each parameter in the following YAML, see .

      [object Object]

4. Run the following command to deliver a single-server single-device job.

   [object Object]

5. Run the following command to check the pod running status.

   [object Object]

   The example output is as follows. If [object Object] appears, the job is running normally.

   [object Object]

   [object Object]

6. View the training results.

   1. Run the following command on any node to view the training results.

      [object Object]

      For example:

      [object Object]

   2. View the training log. If the following content appears, training succeeds.

      [object Object]