Single-Operator API Calling

Single-operator API call refers to directly calling the single-operator API and executing the operator based on C language API. After an operator project is created, based on the project code framework, define the operator prototype, implement the operator on the kernel and tiling on the host, compile and deploy the operator through the project build script, and then call single-operator APIs.

Principles

After the custom operator is compiled and deployed, the single-operator API is automatically generated and can be directly called in an application.

Generally, the single-operator API is defined as a two-phase API. See the following example:

1
2
 
aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor);
aclnnStatus aclnnXxx(void *workspace, uint64_t workspaceSize, aclOpExecutor *executor, aclrtStream stream);

aclnnXxxGetWorkspaceSize/aclnnXxxTensorGetWorkspaceSize is the first-phase API, which is used to compute the workspace size required for the API calling. After obtaining the workspace size, allocate the NPU memory based on the workspace size, and then call the second-phase API aclnnXxx to perform computation. Xxx indicates the operator type passed during operator prototype registration.

The rules for generating input and output parameters of the aclnnXxxGetWorkspaceSize API are as follows:

When the ValueDepend API is used to mark an input as data-dependent during operator prototype registration, an additional API is generated. This API supports one-phase computation in value-dependency scenarios where the input data is empty.

1
 
aclnnStatus aclnnXxxTensorGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor);
In aclnnXxxTensorGetWorkspaceSize, the data types (aclIntArray, aclFloatArray, and aclBoolArray) used in aclnnXxxGetWorkspaceSize are converted to the aclTensor type. The rules for generating other input and output parameters are the same as those in aclnnXxxGetWorkspaceSize. In the following example, x0, x1, and x2 are the inputs declared as data-dependent, and their data types are DT_INT64, DT_BOOL, and DT_FLOAT, respectively. out is the output of the operator. 
1
2
 
aclnnStatus aclnnXxxGetWorkspaceSize(const aclIntArray *x0, const aclBoolArray *x1, const aclFloatArray *x2, aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor);
aclnnStatus aclnnXxxTensorGetWorkspaceSize(const aclTensor *x0, const aclTensor *x1, const aclTensor *x2, aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor);

Pre-operations

Verification Code Project Preparation

The directory structure of the code project is as follows. You can click LINK to obtain a complete sample of the project. 
├── CMakeLists.txt    // Build script
├── main.cpp    // Main process implementation file for single-operator call

Single-Operator Call Sequence

The calling process of the single-operator API is as follows.

This section uses the AddCustom operator as an example to describe how to compile the code logic for single-operator call. Other operators are called in a similar way to the logic AND Add operator. Modify the code based on your actual need.

The following is a code snippet of key steps only, which is not ready to be built or run. After APIs are called, you need to add exception handling branches and record error logs and info logs.

In single-operator API execution mode, .cpp and .h files are automatically generated in the build_out/autogen directory of the build project. When compiling the calling code of the single-operator, ensure that the automatically generated header file for single-operator API execution is included. The following is an example:

1
 
#include "aclnn_add_custom.h"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
 
// 1. Perform initialization.
CHECK_ACL(aclnnInit(nullptr));

// 2. Allocate runtime resources.
const int32_t deviceId = 0;
CHECK_ACL(aclrtSetDevice(deviceId));

// 3. Allocate memory to store the input and output of the operator.
// ......

// 4. Transmit data.
CHECK_ACL(aclrtMemcpy(input0DeviceMem, bufferSize, input0HostData.data(), bufferSize, ACL_MEMCPY_HOST_TO_DEVICE));
CHECK_ACL(aclrtMemcpy(input1DeviceMem, bufferSize, input1HostData.data(), bufferSize, ACL_MEMCPY_HOST_TO_DEVICE));

// 5. Compute the workspace size and allocate memory.
uint64_t workspaceSize = 0;
aclOpExecutor* executor = nullptr;
CHECK_ACL(aclnnAddCustomGetWorkspaceSize(input0, input1, output0, &workspaceSize, &executor));
void* workspaceDeviceMem = nullptr;
if (workspaceSize > 0) {
    CHECK_ACL(aclrtMalloc(&workspaceDeviceMem, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST));
}

// 6. Execute the single-operator.
CHECK_ACL(aclnnAddCustom(workspaceDeviceMem, workspaceSize, executor, stream));

// 7. Synchronization.
CHECK_ACL(aclrtSynchronizeStream(stream));

// 8. Process the output data after the operator is executed, for example, printing data to the screen or writing the data to a file. You can implement this function as required.
// ......

// 9. Destroy runtime allocations.
CHECK_ACL(aclrtResetDevice(deviceId));
// ....

// 10. Perform deinitialization.
CHECK_ACL(aclnnFinalize());

CMakeLists File

After the operator is compiled, the aclnn_xx.h file and dynamic library libcust_opapi.so for single-operator calling are generated. For details about the path, see Pre-operations.

When compiling the operator calling program, add the header file directory for single-operator calling to the search path include_directories of the header file so that the header file can be found. In addition, you need to link the cust_opapi dynamic library and add the directory where libcust_opapi.so is located to the search path link_directories of the library file.

Build and Running

  1. In the development environment, set environment variables and configure the paths of the header files and library files on which the build of single-operator verification program depends. The following is an example of setting environment variables. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann. {arch-os} indicates the architecture and OS of the operating environment. arch indicates the OS architecture, and os indicates the operating system, for example, x86_64-linux or aarch64-linux. 
    export DDK_PATH=${INSTALL_DIR}
export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib
  2. Build the sample project to generate an executable file for single-operator verification.
    1. Go to the directory of the sample project and run the following command in this directory to create a directory (for example, build) for storing the generated executable file. 
      mkdir -p build
    2. Go to the build directory and run the CMake compile command to generate build files.

      Command example:

      cd build
cmake ../src -DCMAKE_SKIP_RPATH=TRUE
    3. Run the following command to generate an executable file: 
      make

      The executable file execute_add_op is generated in the output directory of the project.

  3. Execute the single-operator.
    1. Copy execute_add_op in the output directory of the sample project in the development environment to any directory in the operating environment as the running user (for example, HwHiAiUser).

      Note: If your development environment is the operating environment, skip this step.

    2. Run the execute_add_op file in the operating environment. 
      chmod +x execute_add_op
./execute_add_op

      If "test pass" is displayed, the execution is successful.

Parent topic: Project-based Operator Development