Ascend Docker Runtime

Users who demand containerization support, full-NPU scheduling, static vNPU scheduling, dynamic vNPU scheduling, resumable training, elastic training, inference card fault recovery, or rescheduling upon inference card faults must install Ascend Docker Runtime.
Users who only use resource monitoring do not need to install Ascend Docker Runtime and can skip this chapter directly.

Prerequisites

Before installation, ensure that the user ID of the runc file is 0.
Before installation, in the Containerd scenario, ensure that the user and user group of the /etc/containerd/config.toml file are root; in the Docker scenario or Isula scenario, ensure that the user and user group of the /etc/docker/daemon.json file are root.
Before installation, ensure that the RPATH and RUNPATH of the so files configured in /etc/ld.so.preload do not contain the HDK driver directory with a relative path (you can check this using the readelf -d xxx.so command).

Confirming the Installation Scenario

Currently, only the root user is supported for installing Ascend Docker Runtime. Please select the corresponding installation method based on the actual situation.

Run the following command on the K8s management node to query the node name.

kubectl get node

Example output:

NAME       STATUS   ROLES           AGE   VERSION
ubuntu     Ready    worker          23h   v1.17.3

Check the container runtime of the current node, where node-name is the node name.
- Non-K8s scenario: Run the following command on any node.
```
docker --version      # Docker
containerd --version     # Containerd
```
  - If the output shows Docker version information, the current scenario is Docker scenario.
  - If the output shows Containerd version information, the current scenario is Containerd scenario.
  - If the output contains version information for both Docker and Containerd, determine which container runtime to use for job execution.
- K8s integrated container runtime scenario: Run the following command on the management node.
```
kubectl describe node <node-name> | grep -i runtime
```
  - If the output contains Docker information, this indicates a K8s integrated Docker scenario.
  - If the output contains Containerd information, this indicates a K8s integrated Containerd scenario.

Installing Ascend Docker Runtime in a Docker Scenario

The installation procedure of Ascend Docker Runtime in the K8s-Docker integrated scenario is the same as that in the Docker scenario.

After downloading the software package, go to the path where the software package (run package) is located on all compute nodes.
```
cd <path to run package>
```
Run the following command to add executable permission to the software package.
```
chmod u+x Ascend-docker-runtime_{version}_linux-{arch}.run
```

Run the following command to verify the consistency and integrity of the software package installation file.

./Ascend-docker-runtime_{version}_linux-{arch}.run --check

Example output:

[WARNING]: --check is meaningless for ascend-docker-runtime and will be discarded in the future
Verifying archive integrity...  100%   SHA256 checksums are OK.
...
 All good.

Install Ascend Docker Runtime using the following command.
- To install to the default path, run the following command.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install
```
- To specify installation to a custom path, run the following command. The --install-path parameter specifies the installation path.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-path=<path>
```
- An absolute path must be used for specifying the installation path.
- If the Docker configuration file path is not the default /etc/docker/daemon.json, add the --config-file-path parameter to specify the configuration file path.
The following example output indicates a successful installation.
```
Uncompressing ascend-docker-runtime  100%
[INFO]: installing ascend-docker-runtime
...
[INFO] ascend-docker-runtime install success
```
Run the following command to make Ascend Docker Runtime take effect.
```
systemctl daemon-reload && systemctl restart docker
```
Ascend Device Plugin automatically checks whether Ascend Docker Runtime exists during startup. Therefore, you need to start Ascend Docker Runtime first and then start Ascend Device Plugin. If you start Ascend Device Plugin first and then start Ascend Docker Runtime, you need to restart Ascend Device Plugin by referring to Ascend Device Plugin.

Installing Ascend Docker Runtime in the Containerd Scenario

After the software package is downloaded, go to the path where the software package (run package) is located.
```
cd <path to run package>
```
Run the following command to add executable permission to the software package.
```
chmod u+x Ascend-docker-runtime_{version}_linux-{arch}.run
```

Run the following command to verify the consistency and integrity of the software package installation file.

./Ascend-docker-runtime_{version}_linux-{arch}.run --check

Example output:

[WARNING]: --check is meaningless for ascend-docker-runtime and will be discarded in the future
Verifying archive integrity...  100%   SHA256 checksums are OK.
...
 All good.

Install Ascend Docker Runtime using the following command.
- Install to the default path.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-scene=containerd
```
- To install to a specified path, run the following command. The --install-path parameter specifies the installation path.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-scene=containerd --install-path=<path>
```
  - An absolute path must be used for specifying the installation path.
  - If the Containerd configuration file path is not the default /etc/containerd/config.toml, you need to add the --config-file-path parameter to specify the configuration file path.
The following example output indicates a successful installation.
```
Uncompressing ascend-docker-runtime  100%
[INFO]: installing ascend-docker-runtime
...
[INFO] ascend-docker-runtime install success
```

(Optional) If the installation fails, refer to the following steps to modify the Containerd configuration file.

Open the configuration file.
- Scenario where Containerd has no default configuration file: Run the following commands in sequence to create and modify the configuration file.
```
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
vim /etc/containerd/config.toml
```
- Scenario with an existing Containerd configuration file: Open and modify the configuration file.
```
vim /etc/containerd/config.toml
```

Add ascend runtime and set it as the default runtime. An example is shown below.

Locate the following runc configuration in the configuration file (note that io.containerd.cri.v1.runtime may vary across different Containerd versions; use the actual value):

   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
     runtime_type = 'io.containerd.runc.v2'
     runtime_path = ''
     pod_annotations = []
     container_annotations = []
     privileged_without_host_devices = false
     privileged_without_host_devices_all_devices_allowed = false
     cgroup_writable = false
     base_runtime_spec = ''
     cni_conf_dir = ''
     cni_max_conf_num = 0
     snapshotter = ''
     sandboxer = 'podsandbox'
     io_type = ''

     [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
       BinaryName = ''
       CriuImagePath = ''
       CriuWorkPath = ''
       IoGid = 0
       IoUid = 0
       NoNewKeyring = false
       Root = ''
       ShimCgroup = ''
       SystemdCgroup = true
  ...

Copy the above configuration content, change runc to ascend and set the value of BinaryName to the installation path of the ascend-docker-runtime executable file. Refer to the following (where io.containerd.cri.v1.runtime may vary across different Containerd versions; use the actual value):

   [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.ascend]
     runtime_type = 'io.containerd.runc.v2'
     runtime_path = ''
     pod_annotations = []
     container_annotations = []
     privileged_without_host_devices = false
     privileged_without_host_devices_all_devices_allowed = false
     cgroup_writable = false
     base_runtime_spec = ''
     cni_conf_dir = ''
     cni_max_conf_num = 0
     snapshotter = ''
     sandboxer = 'podsandbox'
     io_type = ''

     [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.ascend.options]
       BinaryName = '/usr/local/Ascend/Ascend-Docker-Runtime/ascend-docker-runtime'
       CriuImagePath = ''
       CriuWorkPath = ''
       IoGid = 0
       IoUid = 0
       NoNewKeyring = false
       Root = ''
       ShimCgroup = ''
       SystemdCgroup = true
  ...

Locate the following configuration item and change the value of default_runtime_name to ascend (where io.containerd.cri.v1.runtime may vary across different Containerd versions; use the actual value):

Before modification:

  [plugins.'io.containerd.cri.v1.runtime'.containerd]
    default_runtime_name = 'runc'
    ignore_blockio_not_enabled_errors = false
    ignore_rdt_not_enabled_errors = false
    ...

After modification:

  [plugins.'io.containerd.cri.v1.runtime'.containerd]
    default_runtime_name = 'ascend'
    ignore_blockio_not_enabled_errors = false
    ignore_rdt_not_enabled_errors = false
  ...

Run the following command to restart Containerd.

systemctl daemon-reload && systemctl restart containerd

Installing Ascend Docker Runtime in the K8s-Containerd Integrated Scenario

After the software package is downloaded, first go to the path where the installation package (run package) is located.
```
cd <path to run package>
```
Run the following command to add executable permission to the software package.
```
chmod u+x Ascend-docker-runtime_{version}_linux-{arch}.run
```

Run the following command to verify the consistency and integrity of the software package installation file.

./Ascend-docker-runtime_{version}_linux-{arch}.run --check

Example output:

[WARNING]: --check is meaningless for ascend-docker-runtime and will be discarded in the future
Verifying archive integrity...  100%   SHA256 checksums are OK.
...
 All good.

You can install Ascend Docker Runtime using the following commands.
- Install to the default path.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-scene=containerd
```
- To install to a specified path, run the following command. The --install-path parameter specifies the installation path.
```
./Ascend-docker-runtime_{version}_linux-{arch}.run --install --install-scene=containerd --install-path=<path>
```
  You must use an absolute path when specifying the installation path.
The following example output indicates a successful installation.
```
Uncompressing ascend-docker-runtime  100%
[INFO]: installing ascend-docker-runtime
...
[INFO] ascend-docker-runtime install success
```
(Optional) If the installation fails, refer to Step 5 in Installing Ascend Docker Runtime in the Containerd Scenario.
If you need to change the container runtime on the node from Docker to Containerd, you must modify the kubelet configuration file kubeadm-flags.env on the node. For details, see the K8s official documentation.
If a Docker service exists, run the following command to stop the corresponding service.
```
systemctl stop docker
```
Run the command to restart Containerd and kubelet. The following is an example output.
```
systemctl daemon-reload && systemctl restart containerd kubelet
```

Parameter Description

Table 1 lists parameters supported by the Ascend Docker Runtime installation commands.

Table 1 Parameters

Parameter	Description
--help \| -h	Query help information.
--install	Install the software package. You can specify the installation path using --install-path=<path>, or install directly to the default path without specifying an installation path.
--install-path=<path>	Specify the installation path. An absolute path must be used as the installation path. If the global configuration file "ascend_docker_runtime_install.info" exists in the environment, the specified installation path must be consistent with the installation path saved in the global configuration file. If you want to change the installation path, you must first uninstall the Ascend Docker Runtime software package from the original path and ensure that the global configuration file "ascend_docker_runtime_install.info" has been deleted. If the Ascend Docker Runtime version prior to 5.0.RC1 was installed via the ToolBox installation package, this file does not exist and does not need to be deleted. If no installation path is specified, it will be installed to the default path "/usr/local/Ascend". If the installation directory is specified using this parameter, the running user must have read and write permissions on the specified installation path.
--install-scene=<scene>	Ascend Docker Runtime installation scenario. The default value is docker, and the value descriptions are as follows. docker: Indicates installation in a Docker (or K8s integration with Docker) scenario. c ontainerd: Indicates installation in a Containerd (or K8s integration with Containerd) scenario. isula: Indicates installation in an iSula container engine scenario. --install-scene cannot be used alone and must be used together with --install, --uninstall, or --upgrade.
--uninstall	Uninstall the software. If an installation path was specified during installation, the installation path must also be specified during uninstallation using the parameter --install-path=<path>.
--upgrade	Upgrade the software. If an installation path was specified during installation, the installation path must also be specified during upgrade using the parameter --install-path=<path>.
--config-file-path=<path>	Configuration file path for Docker or Containerd. If this parameter is not specified, the following default paths are used. Docker: /etc/docker/daemon.json Containerd: /etc/containerd/config.toml
--install-type=<type>	This parameter is only supported when installing or upgrading Ascend Docker Runtime on the following products: Atlas 200 AI acceleration module (RC scenario) Atlas 200I A2 acceleration module (RC scenario) Atlas 200I DK A2 Atlas 200I SoC A1 core board Atlas 500 intelligent edge station (model 3000) Atlas 500 A2 intelligent edge station This parameter is used to set the default mount content for Ascend Docker Runtime, and must be used together with "--install" in the format --install --install-type=<type>. The optional values for <type> are: A200 A200ISoC A200IA2 (supports Atlas 200I A2 acceleration module (RC scenario) and Atlas 200I DK A2) A500 A500A2
--version	Query the Ascend Docker Runtime version.