CANN, PyTorch, Torch-NPU, and ATB Models have been installed in the environment. For details, see MindIE Installation Guide.
[object Object]
- If model initialization fails due to user-defined modifications when ATB Models is used for inference, manually end the process.
- When ATB Models is used for inference, ensure that other users do not have the write permission on the weight path and file.
Currently, ATB Models provides three types of README documents to help you understand the inference process and the features supported by models, and offer basic commissioning and fault locating methods.
Figure 1 ATB Models README document relationship [object Object][object Object]
Table 1 README documents
The following uses LLaMA3-8B as an example to describe how to perform dialog inference and performance test.
Set the environment variable.
Using the
[object Object]package[object Object]Using the
[object Object]package[object Object]
Download model weights from the Hugging Face official website and save the downloaded weight file in
[object Object].Run the following command to change the permission on the weight file:
[object Object](Optional) Convert the weight file format. Currently, only the weight files in .safetensor format can be loaded for ATB Models inference. If the downloaded weight file is in .safetensor format, you do not need to convert the format. If the downloaded weight file is in bin format, perform the following operations:
[object Object]The output result is saved in the same directory where the weight file in .bin format is saved.
Test dialog inference.
[object Object]The
[object Object]script wraps[object Object]. By default, it infers the prompt "What's deep learning?" with a batch size of 1. You can change the inference content in below.[object Object][object Object]Customize the inference content.
You can directly call the
[object Object]script and customize the inference content and inference mode by passing parameters.For example, if the weights in the
[object Object]path are used for 8-device inference of "What's deep learning?" and "Hello World," the batch size is 2.[object Object][object Object]
You can pass token IDs to perform inference.
Create a .py script (for example,
[object Object]) to generate token IDs.[object Object]Run the following command to generate token IDs:
[object Object]Run the following command to start inference. In the following example, the token ID
[object Object]corresponds to the first inference content and the token ID[object Object]corresponds to the second inference content. The inference content is separated by spaces.[object Object]Table 2 Parameters in the
[object Object]script [object Object][object Object][object Object]undefined
[object Object]
Test the performance.
After the
[object Object]environment variable is enabled, the first token, incremental token, and end-to-end inference latency of the model are collected.[object Object]The time consumption result is displayed on the console and saved in the
[object Object]file.[object Object]
CANN, PyTorch, Torch-NPU, ATB Models, MindIE LLM, and MindIE Motor have been installed in the environment. For details, see MindIE Installation Guide.
Set the environment variable. If the installation path is the default path, run the following commands to initialize the environment variables of each component:
Using the
[object Object]package[object Object]Using the
[object Object]package[object Object]
Start serving and send a request.
For details about how to use the MindIE serving, see "Quick Start" > "" in MindIE Motor Development Guide. For details about how to configure serving parameters, see .
In serving configurations, ATB Models is used as the model backend by default.
- Using the
[object Object]package
[object Object]- Using the
[object Object]package
[object Object]For details about serving APIs, see "Serving APIs" in MindIE Motor Development Guide.
You can use an HTTPS client (Linux
[object Object]command, Postman tool, and others) to send HTTPS requests. The following uses Linux[object Object]command as an example. Open a new window and run the following command to send a request:[object Object]- Using the