Deploying RAG SDK in a Container

Preparing for Installation

• The required dependencies have been installed based on Installing Dependencies.
• Before configuring the source, make sure that the installation environment can connect to the network.

Procedure

1. Obtain the base image.
   • You are advised to obtain the RAG SDK image from Ascend Hub. Perform the following steps:
     1. Access Ascend Hub.
     2. Click the Image Version tab.
     3. Click Download following the required version and download the version as prompted. By default, the running user of the downloaded image is HwHiAiUser.
   • If you do not obtain the base image from the Ascend Hub, perform the following steps to prepare an image:
     1. Obtain a Dockerfile. By default, the user specified in the Dockerfile for installing and running RAG SDK and its dependencies is HwHiAiUser. If you need to specify another user, modify the Dockerfile.
     2. Create a package directory in a directory of the same level as the Dockerfile, and store the dependency packages obtained in Installing Dependencies and the RAG SDK package obtained in Downloading the RAG SDK Package in the package directory.
        The following is an example of the file structure. driver is the directory after the npu-driver is installed. The default driver installation path is /usr/local/Ascend/driver/.

        $ tree -L 2
        .
        |-- Dockerfile
        `-- package
            |-- Ascend-cann-<chip_type>-ops_<version>_linux-<arch>.run
            |-- Ascend-cann-toolkit_<version>_linux-<arch>.run
            |-- Ascend-cann-nnal_<version>_linux-<arch>.run
            |-- Ascend-mindxsdk-mxindex_<version>_linux-<arch>.run
            |-- Ascend-mindxsdk-mxrag_<version>_linux-<arch>.run
            `-- driver

     3. Run the following command to build a RAG SDK image:

        docker build -t <Image name>:<Image tag> --build-arg http_proxy=<Proxy> --build-arg https_proxy=<Proxy> --build-arg ARCH=$(uname -m) --build-arg PLATFORM=<chip_type> -f Dockerfile.

        <chip_type> indicates the processor name. You can run the npu-smi info command on the server where the Ascend AI Processor is installed to query the name. Delete the last digit of the queried value of Name, and the remaining part is the value of <chip_type>. If the Atlas 800I A3 SuperPoD Server is used, the value is A3.

2. Run RAG SDK.
   1. Run the following command to start the container where RAG SDK is deployed:

      docker run -e ASCEND_VISIBLE_DEVICES=<device_id>  -itd --name <mxrag_demo> -p <port>:<port> -v <model_dir>:<model_dir>:ro <Image name>:<Image tag> bash

      

      • <device_id>: NPU ID, which starts from 0 by default. If there are multiple values, separate them with commas (,). The NPU used by each container must be exclusively occupied; otherwise, an error will be reported. You can run the ls /dev/davinci* |grep -v /dev/davinci_manager |tr -d /dev/davinci command to query the value.
      • <mxrag_demo>: name of the running container. The default value is mxrag_demo.
      • <port>: port to be mapped.
      • <model_dir>: parent directory for storing the model used by RAG SDK. The value can be /home/data, but cannot be /home or /home/HwHiAiUser. This prevents host file mounting from overwriting the home directory of HwHiAiUser in the container, which would cause RAG SDK exceptions.
      • When the Atlas 800I A2 server uses the Contrastive Language–Image Pre-training (CLIP) model for acceleration, add --device=/dev/dvpp_cmdlist:/dev/dvpp_cmdlist:rw to support DVPP image preprocessing. In addition, ensure that the user running the container has the permission to access /dev/dvpp_cmdlist.
   2. Access the container.

      docker exec -it <mxrag_demo> bash

   3. Run the npu-smi info command to check whether the driver is properly mounted.

      If the value of Health is OK, the current processor is healthy. The following information is only an example.

      +--------------------------------------------------------------------------------------------------------+
      | npu-smi 24.1.rc2                            Version: 24.1.rc2                                          |
      +-------------------------------+-----------------+------------------------------------------------------+
      | NPU     Name                  | Health          | Power(W)     Temp(C)           Hugepages-Usage(page) |
      | Chip    Device                | Bus-Id          | AICore(%)    Memory-Usage(MB)                        |
      +===============================+=================+======================================================+
      | 7       xxx                 | OK              | NA           44                0     / 0             |
      | 0       0                     | 0000:83:00.0    | 0            1851 / 21527                            |
      +===============================+=================+======================================================+
      | 8       xxx                 | OK              | NA           44                0     / 0             |
      | 0       1                     | 0000:84:00.0    | 0            1852 / 21527                            |
      +===============================+=================+======================================================+
      +-------------------------------+-----------------+------------------------------------------------------+
      | NPU     Chip                  | Process id      | Process name             | Process memory(MB)        |
      +===============================+=================+======================================================+
      | No running processes found in NPU 7                                                                    |
      +===============================+=================+======================================================+
      | No running processes found in NPU 8                                                                    |
      +===============================+=================+======================================================+