Model Inference Overview

The model in this chapter refers to the OM model generated by building a trained open-source framework network model based on the CANN Graph Engine (GE). During model build, GE performs operations such as graph optimization, operator scheduling optimization, and weight data rearrangement to further tune the open-source framework network model, so it meets the high-performance requirements in deployment scenarios and runs efficiently on the AI processors.

GE provides the following model build methods:

• Use the Ascend Tensor Compiler (ATC) to build a trained open-source framework network model into an OM model file. For beginners, using ATC commands is recommended. This section uses the ATC commands as an example to describe how to build a model.
• Use the graph development APIs to convert a trained open-source framework network model into an Ascend graph represented by Ascend Intermediate Representation (Ascend IR). For details, see Graph Development.

After the OM model is built, implement model inference based on the model loading and execution APIs provided by GE. The following figure shows the API call sequence.

The key APIs are described as follows:

1. Perform initialization, including calling aclInit to initialize the system and calling aclrtSetDevice to specify the compute device.

   Before using acl APIs to develop an app, you must call aclInit for initialization. Failure to do so may result in errors when internal resources are being initialized, which can then lead to service exceptions.

2. Call the model loading API (for example, aclmdlLoadFromFile) to load the OM model. If the loading is successful, the model ID is returned.
3. Call the model execution API (for example, aclmdlExecute) and specify the model ID to perform inference.

   If the asynchronous model execution API (for example, aclmdlExecuteAsync) is used, call aclrtCreateStream to create a stream before executing the model. A stream is equivalent to a task queue, which ensures that tasks are executed in the sequence in which they enter the queue. After the model is executed, call aclrtSynchronizeStream to wait until all tasks in the stream are complete. After the stream is no longer used, call aclrtDestroyStream to destroy it.

4. Call aclmdlUnload to unload the model.
5. Perform deinitialization, including calling aclrtResetDevice to release resources on the device and calling aclFinalize to free resources used by the acl APIs in the process.