Creating a Container Image Using a Dockerfile (MindSpore)

Prerequisites

Obtain the software packages of the corresponding OS and the Dockerfile and script files required for packaging images by referring to Table 1.

In the software package name, {version} indicates the version number, {arch} indicates the architecture, and {chip_type} indicates the processor type. In CANN 6.3.RC3, 6.2.RC3, and later versions, the message "Do you accept EULA to install CANN (Y/N)" is added to the software package. In the Dockerfile compilation example, the installation command contains the --quiet parameter by default, indicating that EULA is signed by default. You can modify the parameter as required.

The MindSpore software package must match the software of Atlas training product. For details, see MindSpore Installation Guide.

To avoid using software packages that have been tampered with during transmission or storage, download their digital signature files for integrity check while downloading the software packages.

After the software package is downloaded from the Support website, verify its PGP digital signature by referring to the OpenPGP Signature Verification Guide. If the verification fails, do not use the software package, and contact Huawei technical support.

The verification is also required before the installation or update of the software package.

Carriers: Visit https://support.huawei.com/carrier/digitalSignatureAction.

Enterprises: Visit https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054.

• The following uses Ubuntu 18.04 with Python 3.9 and CANN 8.5.0 as an example to describe how to use a Dockerfile to build a container image. Modify the steps as required.
• If MindSpore 2.0.3 or later is used, ubuntu:20.04 must be used.

Procedure

1. Upload the software packages, deep learning framework, host driver installation information file, and driver version information file to the same directory (for example, /home/test) on the server.
   • Ascend-cann-toolkit_{version}_linux-{arch}.run
   • Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
   • mindspore-{version}-cp3x-cp3x-linux_{arch}.whl
   • ascend_install.info
   • version.info
2. Log in to the server as the root user.
3. Perform the following steps to prepare the prebuild.sh file:
   1. Go to the directory where the software packages are stored and run the following command to create the prebuild.sh file:

      vi prebuild.sh

   2. For details about the content to be written, see prebuild.sh compilation example. After writing the content, run the :wq command to save it. The following uses Ubuntu as an example.
4. Perform the following steps to prepare the install_ascend_pkgs.sh file:
   1. Go to the directory where the software packages are stored and run the following command to create the install_ascend_pkgs.sh file:

      vi install_ascend_pkgs.sh

   2. For details about the content to be written, see install_ascend_pkgs.sh compilation example. After writing the content, run the :wq command to save it. The following uses Ubuntu as an example.
5. Perform the following steps to prepare the postbuild.sh file:
   1. Go to the directory where the software packages are stored and run the following command to create the postbuild.sh file:

      vi postbuild.sh

   2. For details about the content to be written, see postbuild.sh compilation example. After writing the content, run the :wq command to save it. The following uses Ubuntu as an example.
6. Perform the following steps to create a Dockerfile:
   1. Go to the directory where the software packages are stored and run the following command to create a Dockerfile:

      vi Dockerfile

   2. For details about the content to be written, see Dockerfile compilation example. After writing the content, run the :wq command to save it. The following uses Ubuntu as an example.
7. Go to the directory where the software packages are stored and run the following command to create a container image. Do not omit the period (.) at the end of the command.

   docker build -t Image name_System architecture:Image tag .

   The following table describes the command parameters.

   Table 2 Command parameters

   Parameter	Description
   -t	Image name.
   Image name_System architecture:Image tag	Image name and tag. Change them based on the actual situation.

   For example:

   docker build -t test_train_arm64:v1.0 .

   If Successfully built xxx is displayed, the image has been created.

8. After the image is created, run the following command to view the image information:

   docker images

   Command output:

   REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
   test_train_arm64          v1.0                d82746acd7f0        27 minutes ago      749MB

9. (Optional) Check whether the base image is available.
   1. Run the following command to mount the driver to the basic image using Ascend Docker Runtime. test_train_arm64:v1.0 is used as an example.

      docker run -it --privileged -e ASCEND_VISIBLE_DEVICES=0 test_train_arm64:v1.0 /bin/bash

   2. Run the following command to check whether the MindSpore is successfully installed in the base image:

      python -c "import mindspore;mindspore.set_context(device_target='Ascend');mindspore.run_check()"

      If the following information is displayed, the MindSpore is successfully installed.

      MindSpore version: version_number
      The result of multiplication calculation is correct, MindSpore has been installed on platform [Ascend] successfully!

Compilation Examples

1. Compilation example of prebuild.sh
   • Compilation example of prebuild.sh for the Ubuntu ARM OS

     #!/bin/bash
     #--------------------------------------------------------------------------------
     # Use the bash syntax to write script code and prepare for the installation, for example, configuring the proxy.
     # This script will be executed before the formal creation process is started.
     #
     # Note: After this script is executed, it will not be automatically cleared. If it does not need to be retained in the image, clear it from the postbuild.sh script.
     #--------------------------------------------------------------------------------
     # DNS settings
     tee /etc/resolv.conf <<- EOF
     nameserver xxx.xxx.xxx.xxx # IP address of the DNS server. You can enter multiple IP addresses as required.
     nameserver xxx.xxx.xxx.xxx
     nameserver xxx.xxx.xxx.xxx
     EOF
     # APT proxy settings
     tee /etc/apt/apt.conf.d/80proxy <<- EOF
     Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";  # IP address and port number of the HTTP proxy server.
     Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";  # IP address and port number of the HTTPS proxy server.
     EOF
     chmod 777 -R /tmp
     rm /var/lib/apt/lists/*
     #APT mirror settings (The following uses Ubuntu 18.04 Arm as an example. Set the information as required.)
     tee /etc/apt/sources.list <<- EOF
     deb http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
     deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
     deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
     deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
     deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
     deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
     deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
     deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
     deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
     deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
     EOF

   • Compilation example of prebuild.sh for the Ubuntu x86_64 OS

     #!/bin/bash
     #--------------------------------------------------------------------------------
     
     # Use the bash syntax to write script code and prepare for the installation, for example, configuring the proxy.
     # This script will be executed before the formal creation process is started.
     #
     # Note: After this script is executed, it will not be automatically cleared. If it does not need to be retained in the image, clear it from the postbuild.sh script.
     #--------------------------------------------------------------------------------
     # APT proxy settings
     tee /etc/apt/apt.conf.d/80proxy <<- EOF
     Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";    # IP address and port number of the HTTP proxy server.
     Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";   # IP address and port number of the HTTPS proxy server.
     EOF
     
     #APT mirror settings (The following uses Ubuntu 18.04 x86_64 as an example. Set the information as required.)
     tee /etc/apt/sources.list <<- EOF
     deb http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
     deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
     deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
     deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
     deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
     deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
     deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
     deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
     deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
     deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
     EOF

2. Compilation example of install_ascend_pkgs.sh

   #!/bin/bash
   #--------------------------------------------------------------------------------
   # Use the bash syntax to write script code and install the Ascend software package.
   #
   # Note: After this script is executed, it will not be automatically cleared. If it does not need to be retained in the image, clear it from the postbuild.sh script.
   #--------------------------------------------------------------------------------
   # Copy the /etc/ascend_install.info file on the host to the current directory before creating the container image.
   cp ascend_install.info /etc/
   mkdir -p /usr/local/Ascend/driver/
   cp version.info /usr/local/Ascend/driver/
   
   # Ascend-cann-toolkit_{version}_linux-{arch}.run
   chmod +x Ascend-cann-toolkit_{version}_linux-{arch}.run
   ./Ascend-cann-toolkit_{version}_linux-{arch}.run --install-path=/usr/local/Ascend/ --install --quiet
   chmod +x Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
   ./Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run --install --quiet
    
   # Only the toolkit package is installed. Therefore, the toolkit package needs to be cleared. When the container is started, the toolkit package is mounted by Ascend Docker.
   rm -f version.info
   rm -rf /usr/local/Ascend/driver/

3. Compilation example of postbuild.sh

   #!/bin/bash
   #--------------------------------------------------------------------------------
   # Use the bash syntax to write the script code and delete the installation packages, scripts, and proxy configurations that do not need to be retained in the container.
   # This script will be run after the formal creation process ends.
   #
   # Note: After this script is run, it is automatically cleared and will not be left in the image. The script and Working Dir are stored in /root.
   #--------------------------------------------------------------------------------
   
   rm -f ascend_install.info
   rm -f prebuild.sh
   rm -f install_ascend_pkgs.sh
   rm -f Dockerfile
   rm -f version.info
   rm -f Ascend-cann-toolkit_{version}_linux-{arch}.run
   rm -f Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
   # Select the package to be deleted based on the installed version.
   rm -f mindspore-{version}-cp3x-cp3x-linux_{arch}.whl
   rm -f /etc/apt/apt.conf.d/80proxy
    
   tee /etc/resolv.conf <<- EOF
   # This file is managed by man:systemd-resolved(8). Do not edit.
   #
   # This is a dynamic resolv.conf file for connecting local clients to the
   # internal DNS stub resolver of systemd-resolved. This file lists all
   # configured search domains.
   #
   # Run "systemd-resolve --status" to see details about the uplink DNS servers
   # currently in use.
   #
   # Third party programs must not access this file directly, but only through the
   # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
   # replace this symlink by a static file or a different symlink.
   #
   # See man:systemd-resolved.service(8) for details about the supported modes of
   # operation for /etc/resolv.conf.
    
   options edns0
    
   nameserver xxx.xxx.xxx.xxx
   nameserver xxx.xxx.xxx.xxx
   EOF

4. Dockerfile compilation sample
   • Dockerfile example of Python 3.9 for the Ubuntu ARM OS

     FROM ubuntu:18.04 
     
     ARG HOST_ASCEND_BASE=/usr/local/Ascend 
     ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh 
     ARG TOOLKIT_PATH=/usr/local/Ascend/ascend-toolkit/latest 
     ARG MINDSPORE_PKG=mindspore-{version}-cp39-cp39-linux_aarch64.whl
     ARG PREBUILD_SH=prebuild.sh 
     ARG POSTBUILD_SH=postbuild.sh 
     WORKDIR /tmp 
     COPY . ./ 
     
     # Trigger prebuild.sh.
     RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH" 
     
     ENV http_proxy http://xxx
     ENV https_proxy http://xxx
     
     
     # System packages
     RUN apt update && \ 
         apt install --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \ 
             libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 -y 
     
     RUN wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
     RUN tar -zxvf Python-3.9.2.tgz && cd Python-3.9.2 && ./configure --prefix=/usr/local/python3.9.2 --enable-shared && make && make install 
     
     RUN ln -s /usr/local/python3.9.2/bin/python3.9 /usr/local/python3.9.2/bin/python && \
         ln -s /usr/local/python3.9.2/bin/pip3.9 /usr/local/python3.9.2/bin/pip
      
     
     # Configure the Python pip mirror.
     RUN mkdir -p ~/.pip \ 
     && echo '[global] \n\ 
     index-url=https://pypi.doubanio.com/simple/\n\ 
     trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf 
      
     # Change the value of PYTHONPATH as required.
     ENV LD_LIBRARY_PATH=/usr/local/python3.9.2/lib:$LD_LIBRARY_PATH
     ENV PATH=/usr/local/python3.9.2/bin:$PATH 
     ENV PYTHONPATH=/usr/local/python3.9.2/lib/python3.9/site-packages:$PYTHONPATH   
     # Create the HwHiAiUser user and owner. The values of UID and GID must be the same as those on the physical machine to avoid generating ownerless files. In the example, the user and the corresponding group are automatically created. The values of UID and GID are both 1000.
     RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser
     
     # Install Python 3.9. For other versions, modify the following commands as required.
     RUN pip install numpy && \ 
         pip install decorator && \ 
         pip install sympy==1.4 && \ 
         pip install cffi==1.12.3 && \ 
         pip install pyyaml && \ 
         pip install pathlib2 && \ 
         pip install grpcio && \ 
         pip install grpcio-tools && \ 
         pip install protobuf && \ 
         pip install scipy && \ 
         pip install requests && \
         pip install kubernetes && \
         pip install attrs && \
         pip install psutil && \
         pip install absl-py
     
     # Ascend package
     RUN umask 0022 && bash $INSTALL_ASCEND_PKGS_SH 
     
     # MindSpore installation
     RUN pip install $MINDSPORE_PKG 
     
     ENV http_proxy "" 
     ENV https_proxy "" 
     
     # Trigger postbuild.sh.
     RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH" && \ 
         rm $POSTBUILD_SH

   • Dockerfile example of Python 3.9 for the Ubuntu x86_64 OS

     FROM ubuntu:18.04 
     
     ARG HOST_ASCEND_BASE=/usr/local/Ascend 
     ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh 
     ARG TOOLKIT_PATH=/usr/local/Ascend/ascend-toolkit/latest  
     ARG MINDSPORE_PKG=mindspore-{version}-cp39-cp39-linux_x86_64.whl
     ARG PREBUILD_SH=prebuild.sh 
     ARG POSTBUILD_SH=postbuild.sh 
     WORKDIR /tmp 
     COPY . ./ 
     
     # Trigger prebuild.sh.
     RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH" 
     
     ENV http_proxy http://xxx
     ENV https_proxy http://xxx
     
     
     # System packages
     RUN apt update && \ 
         apt install --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \ 
             libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 -y 
     
     RUN wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
     RUN tar -zxvf Python-3.9.2.tgz && cd Python-3.9.2 && ./configure --prefix=/usr/local/python3.9.2 --enable-shared && make && make install 
     
     RUN ln -s /usr/local/python3.9.2/bin/python3.9 /usr/local/python3.9.2/bin/python && \
         ln -s /usr/local/python3.9.2/bin/pip3.9 /usr/local/python3.9.2/bin/pip
     
     # Configure the Python pip mirror.
     RUN mkdir -p ~/.pip \ 
     && echo '[global] \n\ 
     index-url=https://pypi.doubanio.com/simple/\n\ 
     trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf 
      
     # Change the value of PYTHONPATH as required.
     ENV LD_LIBRARY_PATH=/usr/local/python3.9.2/lib:$LD_LIBRARY_PATH
     ENV PATH=/usr/local/python3.9.2/bin:$PATH 
     ENV PYTHONPATH=/usr/local/python3.9.2/lib/python3.9/site-packages:$PYTHONPATH  
     # Create the HwHiAiUser user and owner. The values of UID and GID must be the same as those on the physical machine to avoid generating ownerless files. In the example, the user and the corresponding group are automatically created. The values of UID and GID are both 1000.
     RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser
     
     # Install Python 3.9. For other versions, modify the following commands as required.
     RUN pip install numpy && \ 
         pip install decorator && \ 
         pip install sympy==1.4 && \ 
         pip install cffi==1.12.3 && \ 
         pip install pyyaml && \ 
         pip install pathlib2 && \ 
         pip install grpcio && \ 
         pip install grpcio-tools && \ 
         pip install protobuf && \ 
         pip install scipy && \ 
         pip install requests && \
         pip install kubernetes && \
         pip install attrs && \
         pip install psutil && \
         pip install absl-py
     
     # Ascend package
     RUN umask 0022 && bash $INSTALL_ASCEND_PKGS_SH 
     
     # MindSpore installation
     RUN pip install $MINDSPORE_PKG 
     
     ENV http_proxy "" 
     ENV https_proxy "" 
     
     # Trigger postbuild.sh.
     RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH" && \ 
         rm $POSTBUILD_SH