Single-operator API call refers to directly calling the single-operator API and executing the operator based on C language API. After an operator project is created, based on the project code framework, define the operator prototype, implement the operator on the kernel and tiling on the host, compile and deploy the operator through the project build script, and then call single-operator APIs.
After the custom operator is compiled and deployed, the single-operator API is automatically generated and can be directly called in an application.
Generally, the single-operator API is defined as a two-phase API. See the following example:
1 2 |
aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); aclnnStatus aclnnXxx(void *workspace, uint64_t workspaceSize, aclOpExecutor *executor, aclrtStream stream); |
aclnnXxxGetWorkspaceSize/aclnnXxxTensorGetWorkspaceSize is the first-phase API, which is used to compute the workspace size required for the API calling. After obtaining the workspace size, allocate the NPU memory based on the workspace size, and then call the second-phase API aclnnXxx to perform computation. Xxx indicates the operator type passed during operator prototype registration.
The rules for generating input and output parameters of the aclnnXxxGetWorkspaceSize API are as follows:
1
|
aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *xOptional, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); |
1
|
aclnnStatus aclnnXxxGetWorkspaceSize(aclTensor *xRef, ..., uint64_t *workspaceSize, aclOpExecutor **executor); |
1 2 3 4 |
// There is only one output. aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); // There are multiple outputs. aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *src, ..., aclTensor *yOut, aclTensor *y1Out, ..., uint64_t *workspaceSize, aclOpExecutor **executor); |
aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *x, double negativeSlope, aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor);
When the ValueDepend API is used to mark an input as data-dependent during operator prototype registration, an additional API is generated. This API supports one-phase computation in value-dependency scenarios where the input data is empty.
1
|
aclnnStatus aclnnXxxTensorGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); |
1 2 |
aclnnStatus aclnnXxxGetWorkspaceSize(const aclIntArray *x0, const aclBoolArray *x1, const aclFloatArray *x2, aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); aclnnStatus aclnnXxxTensorGetWorkspaceSize(const aclTensor *x0, const aclTensor *x1, const aclTensor *x2, aclTensor *out, uint64_t *workspaceSize, aclOpExecutor **executor); |
"ENABLE_BINARY_PACKAGE": {
"type": "BOOL",
"value": "True"
},
After the operator is built and deployed, the header file aclnn_xx.h and dynamic library libcust_opapi.so for calling a single-operator are generated in the op_api directory under the OPP installation directory.
├── opp // Operator library directory │ ├── vendors // Directory of custom operators │ ├── config.ini │ └── vendor_name1 // Custom operator deployed by the storage vendor. vendor_name is configured during the build of the custom operator installation package. If vendor_name is not configured, the default value customize is used. │ ├── op_api │ │ ├── include │ │ │ └── aclnn_xx.h │ │ └── lib │ │ └── libcust_opapi.so ...
├──input // Directory for storing the input data generated by the script ├──output // Directory for storing the output data and truth value generated during operator execution ├── inc // Header file directory │ ├── common.h // Common method class declaration file, used to read binary files │ ├── operator_desc.h // Operator description declaration file, including the operator input and output, operator type, input description, and output description │ ├── op_runner.h // Operator execution information declaration file, including the numbers and sizes of operator input and output ├── src │ ├── CMakeLists.txt // Build script │ ├── common.cpp // Common function file, used to read binary files │ ├── main.cpp // Entry for single-operator calling │ ├── operator_desc.cpp // File used to construct the input and output description of the operator │ ├── op_runner.cpp // Main process implementation file for single-operator calling ├── scripts │ ├── verify_result.py // Truth value comparison file │ ├── gen_data.py // Script file for generating the input data and truth value │ ├── acl.json // ACL configuration file
The following describes how to compile the main.cpp, op_runner.cpp, and CMakeLists.txt files for single-operator calling.
The calling process of the single-operator API is as follows.
This section uses the AddCustom operator as an example to describe how to compile the code logic for single-operator calling. Other operators are called in a similar way to the logic of calling the Add operator. Modify the code based on your actual need.
The following is a code snippet of key steps only, which is not ready to be built or run. After APIs are called, you need to add exception handling branches and record error logs and info logs.
In single-operator API execution mode, .cpp and .h files are automatically generated in the build_out/autogen directory of the build project. When compiling the calling code of the single-operator, ensure that the automatically generated header file for single-operator API execution is included. The following is an example:
1
|
#include "aclnn_add_custom.h" |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
// 1. Perform initialization. aclRet = aclInit("../scripts/acl.json"); // 2. Allocate runtime resources. int deviceId = 0; aclRet = aclrtSetDevice(deviceId); // Obtain the run mode of the software stack. Different run modes lead to different API call sequences (for example, whether data transfer is required). aclrtRunMode runMode; bool g_isDevice = false; aclError aclRet = aclrtGetRunMode(&runMode); g_isDevice = (runMode == ACL_DEVICE); // 3. Allocate memory to store the input and output of the operator. // ...... // 4. Transmit data. if (aclrtMemcpy(devInputs_[i], size, hostInputs_[i], size, kind) != ACL_SUCCESS) { return false; } // 5. Compute the workspace size and allocate memory. size_t workspaceSize = 0; aclOpExecutor *handle = nullptr; auto ret = aclnnAddCustomGetWorkspaceSize(inputTensor_[0], inputTensor_[1], outputTensor_[0], &workspaceSize, &handle); // ... void *workspace = nullptr; if (workspaceSize != 0) { if (aclrtMalloc(&workspace, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST) != ACL_SUCCESS) { ERROR_LOG("Malloc device memory failed"); } } // 6. Execute the single-operator. if (aclnnAddCustom(workspace, workspaceSize, handle, stream) != ACL_SUCCESS) { (void)aclrtDestroyStream(stream); ERROR_LOG("Execute Operator failed. error code is %d", static_cast<int32_t>(ret)); return false; } // 7. Synchronization. aclrtSynchronizeStream(stream); // 8. Process the output data after the operator is executed, for example, printing data to the screen or writing the data to a file. You can implement this function as required. // ...... // 9. Destroy runtime allocations. aclRet = aclrtResetDevice(deviceId); // .... // 10. Perform deinitialization. aclRet = aclFinalize(); |
After the operator is compiled, the aclnn_xx.h file and dynamic library libcust_opapi.so for single-operator calling are generated. For details about the path, see Pre-operations.
When compiling the operator calling program, add the header file directory for single-operator calling to the search path include_directories of the header file so that the header file can be found. In addition, you need to link the cust_opapi dynamic library and add the directory where libcust_opapi.so is located to the search path link_directories of the library file.
include_directories(
${INC_PATH}/runtime/include
${INC_PATH}/atc/include
../inc
${OP_API_PATH}/include
)target_link_libraries(execute_add_op
ascendcl
cust_opapi
acl_op_compiler
nnopbase
stdc++
)link_directories(
${LIB_PATH}
${LIB_PATH1}
${OP_API_PATH}/lib
)Run the following command in the sample operator project directory:
python3 scripts/gen_data.py
Two data files input_0.bin and input_1.bin with shape (8, 2048) and data type float16 are generated in the current project directory for verifying the AddCustom operator.
A code example is as follows:
import numpy as np
a = np.random.randint(100, size=(8, 2048,)).astype(np.float16)
b = np.random.randint(100, size=(8, 2048,)).astype(np.float16)
a.tofile('input_0.bin')
b.tofile('input_1.bin')
export DDK_PATH=${INSTALL_DIR} export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib
mkdir -p build
Command example:
cd build cmake ../src -DCMAKE_SKIP_RPATH=TRUE
make
The executable file execute_add_op is generated in the output directory of the project.
Note: If your development environment is the operating environment, skip this step.
chmod +x execute_add_op ./execute_add_op
Check the command output:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
[INFO] Set device[0] success [INFO] Get RunMode[1] success [INFO] Init resource success [INFO] Set input success [INFO] Copy input[0] success [INFO] Copy input[1] success [INFO] Create stream success [INFO] Execute aclnnAddCustomGetWorkspaceSize success, workspace size 0 [INFO] Execute aclnnAddCustom success [INFO] Synchronize stream success [INFO] Copy output[0] success [INFO] Write output success [INFO] Run op success [INFO] Reset Device success [INFO] Destroy resource success |
If Run op success is displayed, the execution is successful and the output_z.bin file is generated in the output directory.
Switch to the root directory of the sample project and run the following command:
python3 scripts/verify_result.py output/output_z.bin output/golden.bin
1
|
test pass |
The verification result of the AddCustom operator is correct.