Model Inference Overview

The model in this chapter refers to the OM model generated by compiling the trained open source network model based on the CANN Graph Engine (GE). When building a model, GE performs operations such as graph optimization, operator scheduling optimization, and weight data rearrangement to further optimize the network model of the open-source framework, meet high performance requirements in deployment scenarios, and ensure that the model runs efficiently on AI processors.

GE provides the following model build modes:

• Use the Ascend Tensor Compiler (ATC) to compile the trained open source network model into an OM model file. The ATC command is suitable for beginners. This section uses the ATC command as an example to describe how to build a model.
• Convert the open source network model trained using the graph development API into a computational graph (Ascend Graph) represented by Ascend IR (Ascend Intermediate Representation). For details, see Graph Development.

After the OM model is compiled, implement model inference based on the model loading and execution APIs provided by GE. The following figure shows the API call sequence.

The key APIs are described as follows:

1. Initialization includes calling aclInit to initialize the system and calling aclrtSetDevice to specify a compute device.

   Before using the acl API to develop an application, you must call the aclInit API to initialize the application. Otherwise, errors may occur during the initialization of internal system resources, causing exceptions in other services.

2. Call the model loading API (for example, aclmdlLoadFromFile) to load the OM model. If the loading is successful, the model ID is returned.
3. Call the model execution API (for example, aclmdlExecute) and specify the model ID to perform inference.

   If the asynchronous model execution API (for example, aclmdlExecuteAsync) is used, call aclrtCreateStream to create a stream before executing the model. A stream is equivalent to a task queue, ensuring that tasks are executed in sequence based on the sequence in which they enter the queue. After the model is executed, call aclrtSynchronizeStream to wait until all tasks in the stream are complete. After the stream is used, call aclrtDestroyStream to destroy the stream.

4. aclmdlUnload: unloads a model.
5. Deinitialization includes calling aclrtResetDevice to release resources on the device and calling aclFinalize to release resources used by the ACL APIs in the process.