Concepts and Usage

Basic Concepts

The APIs in this section manipulate the operator binary, kernel function, kernel function parameter list, and parameters. The following figure shows the relationships between them.

• Operator binary: After the operator source code is built, the operator binary file *.o is obtained. For a built-in CANN operator, you can obtain the operator binary file from the operator binary package (Ascend-cann-kernels-*.run). For a custom operator, you can obtain the operator binary file after building the operator and releasing the binary file. For details about how to develop and build a custom operator, see Ascend C Operator Development.
• Kernel function: It is an entry function for implementing an operator on the device. You can use the syntax extension of a C/C++ function to write the running code on the device. You can also perform data access and computing operations in the kernel function to implement all functions of the operator.

API Call Sequence for Kernel Loading and Execution

The major steps are as follows:

1. Call acl.init for initialization.
2. Allocate runtime resources, including calling the acl.rt.set_device API to specify a device for computation and calling the acl.rt.create_stream API to create a stream.

   For details, see Runtime Resource Allocation and Deallocation.

3. Call the acl.rt.binary_load_from_file API to load the operator binary file.
4. Call acl.rt.binary_get_function to obtain the kernel function handle.
5. Operate the parameter list according to the kernel function handle. The operations are as follows:
   1. Initialize the parameter list.

      Currently, the memory can be managed by the system (by calling acl.rt.kernel_args_init) or by users (by calling acl.rt.kernel_args_init_by_user_mem).

   2. Append parameters and update parameter values.

      The kernel function parameter list contains parameters of different types, such as pointer, placeholder, and uint8_t parameters.

      • Pointer parameters: The values indicate the device memory address. Generally, the input and output of an operator are of this type. You need to call the device memory allocation API (such as the acl.rt.malloc API) in advance to allocate memory and copy data to the device.
      • Placeholder: A placeholder is also a pointer parameter. The difference is that you do not need to manually copy the parameter data to the device. Instead, this operation is completed by the Runtime. The Runtime does not fill the actual device address when a parameter is appended, but fills it only during kernel launch. That is where the placeholder comes in. For non-input and non-output parameters of an operator, you can use a placeholder to combine the host-to-device copies of small data (< 2 KB recommended) into one copy during kernel launch, thus reducing the number of copy operations and improving performance.

      

      You can call different parameter appending APIs for different types of parameters.

      • For placeholder parameters, the associated memory must be placed after all parameters. Therefore, when appending parameters, call the acl.rt.kernel_args_append_place_holder API to reserve a placeholder. After all parameters are appended, call the acl.rt.kernel_args_get_place_holder_buffer API to obtain the memory address pointed to by the placeholder. You can manage the data in the memory based on the obtained memory address.
      • For non-placeholder parameters (such as pointer parameters and uint8_t parameters), call the acl.rt.kernel_args_append API to append the parameter values set by the user to the parameter data area pointed to by argsHandle. To update the parameter value, call the acl.rt.kernel_args_para_update API.

      Note that the kernel function parameter list may contain multiple parameters, and parameters of different types may appear alternately. Therefore, you need to append parameters from left to right according to the parameter sequence in the parameter list. A maximum of 128 parameters can be appended.

   3. End the parameter appending and parameter value update.

      After all parameters are appended, call the acl.rt.kernel_args_finalize API to indicate that the parameter assembly is complete. However, after the acl.rt.kernel_args_finalize API is called, the parameter value can be updated. After the update, the acl.rt.kernel_args_finalize API needs to be called again.

6. Call the acl.rt.launch_kernel_with_config API to launch the kernel and start the compute task of the corresponding operator.
7. Call the acl.rt.binary_unload API to unload the operator binary file.
8. Releases runtime resources, including calling the acl.rt.destroy_stream API to release streams and calling the acl.rt.reset_device API to release resources on the device.

   For details, see Runtime Resource Allocation and Deallocation.

9. Call the acl.finalize API for deinitialization.