Overview
In single-operator model execution, operator execution is based on graph IR. First, the operator is compiled (for example, the ATC tool is used to compile the single-operator description file defined by Ascend IR into an operator OM model file). Then, an acl API (for example, aclopSetModelDir) is called to load the operator model. Finally, an acl API (for example, aclopExecuteV2) is called to execute the operator.
- Compile an operator.
Operators can be compiled in either of the following modes:
- After an operator is compiled, the operator data is saved in the OM file.
In this mode, you need to use the ATC tool to compile the single-operator definition file (.json) into an offline model adapted to the AI processor (OM file). For details, see ATC.
After the compilation, perform 2, 3, 4, 5, 6, and 7 in sequence.
- After an operator is compiled, the operator data is saved in memory.
In this mode, you need to call acl APIs as required.
- For the operators that will be executed for multiple times, you are advised to call aclopCompile to compile the operators. After the compilation, perform 3, 4, 5, 6, and 7 in sequence.
- For the operators that will be compiled and executed for the same number of times, you are advised to perform 3 and then call aclopCompileAndExecute. After the compilation, perform 6 and 7 in sequence.
- After an operator is compiled, the operator data is saved in the OM file.
- Load the operator model file.You can use either of the following methods:
- Call aclopSetModelDir to set the directory of the single-operator OM model file.
- Call aclopLoad to load the single-operator model data from memory. The memory is managed by the user. The single-operator model data refers to the data that is loaded to the memory from the single-operator OM file.
- Call aclrtMalloc to allocate device memory to store the input and output data of the operator.
To transfer data from the host to the device, call aclrtMemcpy (synchronous mode) or aclrtMemcpyAsync (asynchronous mode) to copy the memory.
- In the dynamic-shape scenario, if the output shape of an operator cannot be determined, you need to infer or estimate the output shape of the operator before executing the operator.
You need to call aclopInferShape, aclGetTensorDescNumDims, aclGetTensorDescDimV2, and aclGetTensorDescDimRange to infer or estimate the output shape of the operator as the input of the operator execution API aclopExecuteV2.
- Execute the operator.
- Operators encapsulated as acl APIs, including the GEMM operator (see the CBLAS API) and Cast operator, can be executed in either of the following ways:
- Non-handle mode: Call APIs whose names do not contain keyword "Handle", for example, aclblasGemmEx (with the GEMM operator encapsulated) and aclopCast (with the Cast operator encapsulated).
- Handle mode: Call APIs whose names contain keyword "Handle", for example, aclblasCreateHandleForGemmEx and aclopCreateHandleForCast. Also call aclopExecWithHandle to execute the operators.
- Operators that are not encapsulated as acl APIs can be executed in either of the following ways:
- Non-handle mode: Call aclopExecuteV2.
- Handle mode: Call aclopCreateHandle to create a handle, and then call aclopExecWithHandle.
If an operator is executed in non-handle mode, the system matches the model in the memory based on the operator description in every execution.
When an operator is executed in handle mode, the system matches the operator description information with the model in the memory and caches the information in the handle. Each time the operator is executed, the operator and model do not need to be matched repeatedly. Therefore, when the same operator is executed for multiple times, the efficiency is higher. However, this mode does not support dynamic-shape operators. After the handle is used, aclopDestroyHandle needs to be called to release the handle.
- Operators encapsulated as acl APIs, including the GEMM operator (see the CBLAS API) and Cast operator, can be executed in either of the following ways:
- Call aclrtSynchronizeStream to block the application until all tasks in the specified stream are complete.
- Call aclrtFree to free the memory.
Call aclrtMemcpy (synchronous mode) or aclrtMemcpyAsync (asynchronous mode) to transfer data from the device to the host using memory copy and then free the memory.