Creating an Inference Image

This section describes how to create an image that contains an inference program.

A ResNet-50 image classification sample is used as an example. The sample has been compiled and contains a model. The sample package is vpc_resnet50_imagenet_classification.tar.gz, and the sample executable file is main. In the decompressed vpc_resnet50_imagenet_classification/out directory, upload the sample package to any directory on the server, for example, /home/test.

Typical Service Scenarios

Some applications require the NPU computing power of edge devices. The following describes how to create a corresponding inference image, which is for reference only.

The image must have the following features:

The image is run by the HwHiAiUser user.
When deploying containers, you do not need to configure the privileged container, capability set, and host network.

Prerequisites

Prepare the container OS image by yourself.
Prepare the offline inference engine package and service inference program package according to the following table.

**Table 1** Required software
Software Package	Description	How to Obtain
Ascend-cann-toolkit_{version}_linux-aarch64.run Ascend-cann-310b-ops_{version}_linux-aarch64.run	Offline inference engine package. {version} indicates the software package version. Version 8.5.0 is recommended.	Link
Dockerfile	Required for creating an image.	Prepared by users.
Service inference program package	It is a set of service inference programs, including model files, inference code, and configuration files. The files can be in .tar or .tar.gz format. Modify the commands in install.sh for installing the service inference programs as required. NOTE: The running user in the container must have the required permissions on the service inference program package.	Prepared by users.
slog.conf	Log configuration file. Each inference image requires an independent configuration file. Therefore, you need to copy the configuration file to the image when creating an inference image.	Obtain the configuration file from the image creation environment or user operating environment.
install.sh	Installation script of service inference programs.	Prepared by users.
run.sh	Running script of service inference programs.	Prepared by users.

Verifying the Digital Signature

To prevent a software package from being maliciously tampered with during transfer or storage, download also the corresponding digital signature file for integrity verification while downloading the software package.

Click PGP Verify to obtain the tool package, decompress it and verify the PGP digital signature. For details, see OpenPGP Signature Verification Guide. If the verification fails, do not use the software package. Visit the support website to get help from the community or submit a service ticket.

Procedure

Upload the software packages and scripts to the same directory (for example, /home/test).
- Ascend-cann-toolkit_{version}_linux-aarch64.run
- Ascend-cann-310b-ops_{version}_linux-aarch64.run
- Service inference program package
- Dockerfile
- slog.conf
- install.sh
- run.sh

Perform the following steps to create a Dockerfile.

Log in as the root user and run the following commands in sequence to query and record the UIDs and GIDs (which will be used in the Dockerfile file) of the HwHiAiUser, HwBaseUser, and HwDmUser users in the image environment:
```
id HwHiAiUser
id HwBaseUser
id HwDmUser
```
Go to the software package upload directory in Step 1 and run the following command to create a Dockerfile:
```
vi Dockerfile
```

Enter the following content and run the :wq command to save the modification. (The Ubuntu Arm OS is used as an example, and the content is only an example. Perform secondary development as required.)

# Container operating system and tag. Change them based on the actual situation.
FROM ubuntu:22.04

ARG TOOLKIT_PKG
ARG OPS_PKG
ARG DIST_PKG
# In this document, the /usr/local/Ascend directory is used as the CANN installation directory. If you want to install CANN in another directory, change the directory to the desired one.
ARG ASCEND_BASE=/usr/local/Ascend
WORKDIR /home/AscendWork
COPY $TOOLKIT_PKG .
COPY $OPS_PKG .
COPY $DIST_PKG .
COPY install.sh .

# The inference program needs to use the bottom-layer driver. The running of the bottom-layer driver depends on three users, HwHiAiUser, HwDmUser, and HwBaseUser.
# Create a user and user group for running the inference application. For example, the UIDs and GIDs of HwHiAiUse, HwDmUser, and HwBaseUser are 1000, 1101, and 1102, respectively.
RUN umask 0022 && \
    groupadd  HwHiAiUser -g 1000 && \
    useradd -d /home/HwHiAiUser -u 1000 -g 1000 -m -s /bin/bash HwHiAiUser && \
    groupadd HwDmUser -g 1101 && \
    useradd -d /home/HwDmUser -u 1101 -g 1101 -m -s /bin/bash HwDmUser && \
    usermod -aG HwDmUser HwHiAiUser && \
    groupadd HwBaseUser -g 1102 && \
    useradd -d /home/HwBaseUser -u 1102 -g 1102 -m -s /bin/bash HwBaseUser && \
    usermod -aG HwBaseUser HwHiAiUser

# Install Python on which CANN depends.
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y python3 python3-pip ca-certificates

# Install CANN and decompress the inference program.
RUN chmod +x $TOOLKIT_PKG && \
    chmod +x $OPS_PKG && \
    ./$TOOLKIT_PKG --install --install-path=$ASCEND_BASE --install-for-all --whitelist=nnrt --quiet --force && \
    ./$OPS_PKG --quiet --install --install-path=$ASCEND_BASE --install-for-all --whitelist=nnrt --quiet --force && \
    sh install.sh && \
    chown -R HwHiAiUser:HwHiAiUser /home/AscendWork/ && \
    rm $TOOLKIT_PKG && \
    rm $OPS_PKG && \
    rm $DIST_PKG && \
    rm install.sh

ENV LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64:/usr/lib64
ENV LD_PRELOAD=/lib/aarch64-linux-gnu/libc.so.6

RUN ln -sf /lib /lib64 && \
    mkdir /var/dmp && \
    mkdir /usr/slog && \
    chown HwHiAiUser:HwHiAiUser /usr/slog && \
    chown HwHiAiUser:HwHiAiUser /var/dmp

# Copy the log configuration file.
COPY --chown=HwHiAiUser:HwHiAiUser slog.conf /etc

COPY --chown=HwHiAiUser:HwHiAiUser run.sh /home/AscendWork/run.sh
RUN chmod 640 /etc/slog.conf && \
    chmod +x /home/AscendWork/run.sh
	
# If you need to use the AI CPU operator, delete the comment tag (#) from the following commands:
#RUN cp /usr/local/Ascend/ascend-toolkit/latest/opp/Ascend/aicpu/Ascend-aicpu_syskernels.tar.gz /home/HwHiAiUser/ && \
#    rm -rf /usr/local/Ascend/ascend-toolkit/latest/opp/Ascend/aicpu/Ascend-aicpu_syskernels.tar.gz && \
#    echo $(wc -c /home/HwHiAiUser/Ascend-aicpu_syskernels.tar.gz|awk ' {print$1} ') > /home/HwHiAiUser/aicpu_package_install.info && \
#    tail -c +8449 /home/HwHiAiUser/Ascend-aicpu_syskernels.tar.gz > /home/HwHiAiUser/aicpu.tar.gz && \
#    rm -rf /home/HwHiAiUser/Ascend-aicpu_syskernels.tar.gz && \
#    chown HwHiAiUser:HwHiAiUser /home/HwHiAiUser/aicpu.tar.gz && \
#    mkdir -p /home/HwHiAiUser/aicpu_kernels
#RUN tar -xvf /home/HwHiAiUser/aicpu.tar.gz -C /home/HwHiAiUser/ 2>/dev/null;exit 0
#RUN rm -rf /home/HwHiAiUser/aicpu.tar.gz && \
#    mv /home/HwHiAiUser/aicpu_kernels_device/* /home/HwHiAiUser/aicpu_kernels/ && \
#    chown -R HwHiAiUser:HwHiAiUser /home/HwHiAiUser/

# If you do not need to use the AI CPU operator, delete the comment tag (#) from the following commands:
#RUN rm -rf /usr/local/Ascend/ascend-toolkit/latest/opp/Ascend/aicpu/Ascend-aicpu_syskernels.tar.gz && \
#    chown -R HwHiAiUser:HwHiAiUser /home/HwHiAiUser/

USER 1000
CMD bash /home/AscendWork/run.sh

After creating the Dockerfile, run the following command to change the permission on the Dockerfile:
```
chmod 600 Dockerfile
```

Compilation example of install.sh:

cd /home/AscendWork
# Decompress the service inference program package based on the package format.
tar -xzvf vpc_resnet50_imagenet_classification.tar.gz

Compilation example of run.sh:

#!/bin/bash
mkdir /dev/shm/dmp
nohup /var/dmp_daemon -I -M -U 8087 >&/dev/null &
/var/slogd -d
# Import CANN environment variables. Change the path to the actual CANN installation path.
source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh
# Go to the directory where the executable file of the service inference programs is located. Use the actual path.
cd /home/AscendWork/vpc_resnet50_imagenet_classification/out
# Run the executable file. You need to modify the file according to the actual situation.
./main

Go to the directory where the software packages are stored and run the following command to create a container image:

docker build -t image-name:tag --build-arg TOOLKIT_PKG=toolkit-name --build-arg OPS_PKG=ops-name --build-arg DIST_PKG=distpackage-name .

If the message "Successfully built xxx" is displayed, the image is successfully built. For details about the command, see Table 2.

Do not omit the period (.) at the end of the command.

**Table 2** Parameters in the commands
Parameter	Description
image-name:tag	Specifies the image name and tag. Set this parameter as required.
TOOLKIT_PKG	toolkit-name specifies the name of the offline inference engine package. Replace it with the actual one. Do not omit the file name extension.
OPS_PKG	ops-name specifies the name of the offline inference engine operator package. Replace it with the actual one. Do not omit the file name extension.
DIST_PKG	distpackage-name specifies the name of the service inference program package. Do not omit the file name extension. Replace it with the actual one.

Run the following command to save the container image to the drive:
```
docker save image-name:tag |gzip -c > image-name.tar.gz
```

Parent topic: Common Operations