nnopbase API List

Both Ascend C custom operators and CANN built-in operators (such as Transformer and NN APIs) can call underlying operators through aclnn APIs (also called Level 2 APIs).

When developing aclnn APIs, refer to this section to use the nnopbase APIs required for development, including framework capability APIs and basic tensor APIs (also called Level 0 APIs). The source code of the nnopbase APIs is already available in the opbase repository (cann/opbase).

Framework Capability APIs
Framework scheduling and management APIs for implementing aclnn APIs, such as operator executor (opExecutor) and data type/format/shape operations, as shown in Table 1. In addition, common classes and macros are included. For details, see Table 2 and Table 3. The source code of these APIs has been open-sourced in CANN/opbase.
Basic Tensor APIs
APIs for implementing the aclnn APIs, such as those for tensor data type conversion and shape reconstruction. For details, see Table 4. The source code implementation of these APIs has been open-sourced in CANN/ops-math.

API List

Header file: When using the APIs in this section, include the dependent header files as required. The header files are stored in the ${INSTALL_DIR}/include directory. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.

**Table 1** Framework capability APIs
API Category	Description	Header File
bfloat16	Describes the implementation classes of the bfloat16 type on the CPU.	aclnn/opdev/bfloat16.h
common_types	Describes basic aclnn data structures such as aclTensor and aclScalar.	aclnn/opdev/common_types.h
data_type_utils	Provides basic APIs related to data types, for example, the API for obtaining the size of a specified data type.	aclnn/opdev/data_type_utils.h
fast_vector	Describes the FastVector type, an efficient vector data structure implemented in aclnn. Note: The APIs defined in this header file are reserved and can be ignored.	aclnn/opdev/fast_vector.h
format_utils	Provides basic APIs related to the format.	aclnn/opdev/format_utils.h
fp16_t	Describes the implementation classes of the float16 type on the CPU.	aclnn/opdev/fp16_t.h
float4_e1m2	Describes the implementation classes of the Float4E1M2 type on the CPU.	aclnn/opdev/float4_e1m2.h
float4_e2m1	Describes the implementation classes of the Float4E2M1 type on the CPU.	aclnn/opdev/float4_e2m1.h
float6_e2m3	Describes the implementation classes of the Float6E2M3 type on the CPU.	aclnn/opdev/float6_e2m3.h
float6_e3m2	Describes the implementation classes of the Float6E3M2 type on the CPU.	aclnn/opdev/float6_e3m2.h
float8_e4m3fn	Describes the implementation classes of the Float8E4M3FN type on the CPU.	aclnn/opdev/float8_e4m3fn.h
float8_e5m2	Describes the implementation classes of the Float8E5M2 type on the CPU.	aclnn/opdev/float8_e5m2.h
float8_e8m0	Describes the implementation classes of the Float8E8M0 type on the CPU.	aclnn/opdev/float8_e8m0.h
hifloat4	Describes the implementation classes of the HiFloat4 type on the CPU.	aclnn/opdev/hifloat4.h
hifloat8	Describes the implementation classes of the HiFloat8 type on the CPU.	aclnn/opdev/hifloat8.h
framework_op	Describes the framework's capability of copying data from the host to the device.	aclnn/opdev/framework_op.h
make_op_executor	Provides the macro declaration for initializing aclOpExecutor. Note: The APIs defined in this header file are reserved and can be ignored.	aclnn/opdev/make_op_executor.h
object	Describes the base class Object of basic data structures such as aclTensor in aclnn. It is used to overload and implement the new and delete methods.	aclnn/opdev/object.h
op_arg_def	Describes the OpArgContext class and provides macro declarations such as OP_INPUT.	aclnn/opdev/op_arg_def.h
op_cache	Describes OpExecCache and related classes, which are used to cache aclnn and improve runtime performance.	aclnn/opdev/op_cache.h
op_cache_container	Describes the aclnn cache container with least recently used (LRU) mechanism.	aclnn/opdev/op_cache_container.h
op_config	Provides configurations related to operator running, such as the deterministic computation switch.	aclnn/opdev/op_config.h
op_def	Defines basic enumerations and constants, such as the precision mode OpImplMode.	aclnn/opdev/op_def.h
op_dfx	Describes the DfxGuard class used for API printing and profiling reporting.	aclnn/opdev/op_dfx.h
aclnn Return Codes	Defines the aclnn error codes.	aclnn/opdev/op_errno.h
op_executor	Describes the aclOpExecutor class.	aclnn/opdev/op_executor.h
op_log	Defines the log printing macro in aclnn.	aclnn/opdev/op_log.h
platform	Describes the PlatformInfo class that stores SoC platform information.	aclnn/opdev/platform.h
pool_allocator	Describes the PoolAllocator class that implements the CPU memory pool in aclnn.	aclnn/opdev/pool_allocator.h
shape_utils	Provides basic operations related to the shape, such as shape printing.	aclnn/opdev/shape_utils.h
small_vector	Describes the SmallVector class, an efficient vector data structure implemented in aclnn for a small known data size.	aclnn/opdev/small_vector.h
tensor_view_utils	Provides basic operations on the View class, for example, determining whether aclTensor is contiguous.	aclnn/opdev/tensor_view_utils.h
data_type_utils	Provides basic APIs related to the data type, for example, checking whether a specified data type is an integer.	aclnn/opdev/op_common/data_type_utils.h
aicpu_args_handler	Provides the handling logic of concatenated computing tasks related to the AI CPU, for example, combining arguments related to computing tasks.	aclnn/opdev/aicpu/aicpu_args_handler.h
aicpu_ext_info_handle	Provides the handling logic of extended arguments of computing tasks related to the AI CPU, for example, the API for combining and parsing extended arguments.	aclnn/opdev/aicpu/aicpu_ext_info_handle.h
aicpu_task	Provides logic for setting and delivering AI CPU tasks, for example, setting an AI CPU operator to be called and setting the operator input and output.	aclnn/opdev/aicpu/aicpu_task.h
aicpu_utils	Describes the common APIs required by AI CPU tasks.	aclnn/opdev/aicpu/aicpu_utils.h

**Table 2** Common macros
Macro	Description	Header File
DFX_IN	Packs all API input parameters on the host in L2_DFX_PHASE_1.	aclnn/opdev/op_dfx.h
DFX_OUT	Packs all API output parameters on the host in L2_DFX_PHASE_1.
L0_DFX	Must be used in the L0 API on the host to print the API and input parameters in the L0 API.
L2_DFX_PHASE_1	Must be called at the beginning of the first-phase API to print the API and input parameters in the first phase.
L2_DFX_PHASE_2	Must be called at the beginning of the second-phase API for API printing.
OP_TYPE_REGISTER	Must be used at the beginning of the L0 API to register the L0 operator.
OP_ATTR	Packs operator attribute parameters in ADD_TO_LAUNCHER_LIST_AICORE.	aclnn/opdev/op_arg_def.h
OP_EMPTY_ARG	Holds the place for an empty input or output in ADD_TO_LAUNCHER_LIST_AICORE.
OP_INPUT	Packs the operator input aclTensor in ADD_TO_LAUNCHER_LIST_AICORE.
OP_MODE	Packs operator runtime options in ADD_TO_LAUNCHER_LIST_AICORE, such as whether to enable HF32.
OP_OUTPUT	Packs the operator output aclTensor in ADD_TO_LAUNCHER_LIST_AICORE.
OP_OUTSHAPE	Sets the aclTensor for storing the output shape for the third type of operators in ADD_TO_LAUNCHER_LIST_AICORE.
OP_OPTION	Packs the precision mode specified by the operator in ADD_TO_LAUNCHER_LIST_AICORE.
OP_WORKSPACE	Packs the workspace parameter explicitly specified by the operator in ADD_TO_LAUNCHER_LIST_AICORE.
CREATE_EXECUTOR	Creates a UniqueExecutor object, which is the factory class of aclOpExecutor.	aclnn/opdev/make_op_executor.h
INFER_SHAPE	Runs the infershape function of a specified operator to infer the output shape.
ADD_TO_LAUNCHER_LIST_AICORE	Creates an execution task for an AI Core operator and adds the task to the aclOpExecutor execution queue. This API is executed in the second phase.
OP_ATTR_NAMES	Packs the character type attributes of an AI CPU operator. It is a vector of string type.	aclnn/opdev/aicpu/aicpu_task.h
ADD_TO_LAUNCHER_LIST_AICPU	Creates an execution task for an AI CPU operator and adds the task to the aclOpExecutor execution queue. This API is executed in the second phase.	aclnn/opdev/aicpu/aicpu_task.h

**Table 3** Common classes and structs
Class/Struct	Description	Header File
aclOpExecutor	Indicates the operator executor, which records the context structure of the API running information on the host, such as the computational graph during the execution of an L2 API, launch subtask of an L0 operator, and workspace address and size. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see op_executor.	aclnn/opdev/op_executor.h
aclTensor	Indicates a tensor object, including the shape, data type, format, and address of the tensor. The data can be stored on the host or device. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.	aclnn/opdev/common_type.h
aclScalar	Indicates a scalar object. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclTensorList	Indicates a list object consisting of a group of aclTensor types. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclScalarList	Indicates a list object consisting of a group of aclScalar types. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclBoolArray	Indicates an array object of the Boolean type. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclIntArray	Indicates an array object of the int64_t type. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclFloatArray	Indicates an array object of the fp32 type. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclFp16Array	Indicates an array object of the fp16 type. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
aclBf16Array	Indicates an array object of the bf16 type. The data is generally stored on the host. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see common_types.
SmallVector	This class uses the internal memory pool to implement the vector container. The basic functions of this class are the same as those of the std::vector container in the C++ standard library. Not every capacity expansion requires memory allocation so that the performance is not affected. The member variables defined by this class are private and can be ignored. For details about the defined member functions, see small_vector.	aclnn/opdev/small_vector.h
OpExecMode	Indicates the enumeration class for the operator run mode. For details, see OpExecMode.	aclnn/opdev/op_def.h
OpImplMode	Indicates the enumeration class for the operator precision modes. For details, see OpImplMode.	aclnn/opdev/op_def.h

Instructions for the SmallVector class:

op::FVector: SmallVector whose storage capacity length is 8.

namespace op {
template<typename T, size_t N = 8>
using FVector = op::internal::SmallVector<T, N, op::internal::PoolAllocator<T>>;
}

op:Strides: FVector whose storage capacity length is 25. It has int64_t elements and stores the stride information.

namespace op {
constexpr uint64_t MAX_DIM_NUM = 25;
using Strides = FVector<int64_t, MAX_DIM_NUM>;
}

op::ShapeVector: FVector whose storage capacity length is 25. It has int64_t elements and stores the shape information.

namespace op {
constexpr uint64_t MAX_DIM_NUM = 25;
using ShapeVector = FVector<int64_t, MAX_DIM_NUM>;
}

**Table 4** Basic tensor APIs
API	Description	Header File
Cast	Converts an input tensor to the specified data type.	aclnn_kernels/cast.h
Contiguous	Converts a non-contiguous tensor into a contiguous one.	aclnn_kernels/contiguous.h
ViewCopy	Moves a contiguous tensor to a contiguous or non-contiguous one.	aclnn_kernels/contiguous.h
Pad	Pads dimensions of an input tensor based on paddings. The padding value is 0.	aclnn_kernels/pad.h
Reshape	Reshapes the input tensor x.	aclnn_kernels/reshape.h
Slice	Obtains required tensor slices.	aclnn_kernels/slice.h
Transpose	Transposes the shape of input tensor x according to the permutation of dimensions (perm) and outputs the result.	aclnn_kernels/transpose.h
TransData	Converts the format of an input tensor to the specified dstPrimaryFormat.	aclnn_kernels/transdata.h
TransDataSpecial	Converts the format of an input tensor to the specified dstPrimaryFormat. This API is similar to TransData.
ReFormat	Reformats the input tensor x to the destination format without changing the dimensions.
IsNullptr	Checks whether the input pointer is null.	aclnn_kernels/common/op_error_check.h

Basic Concepts

Level-0 APIs

Calling L0 APIs is to call single-kernel APIs on the host. This level provides fine-grained APIs (single-kernel delivery), basic structures (such as tensor definition) for operator API development, and common basic capabilities (such as workspace reuse and engine scheduling). Upper-layer applications or L2 APIs can quickly assemble L0 APIs to implement high-performance computing.

The return type of an L0 API is a tensor type structure, such as aclTensor*, std::tuple<aclTensor*, aclTensor*&gt, or aclTensorList*. The last parameter is fixed at aclOpExecutor *executor, and the type and name cannot be changed. The following is an example:

aclTensor* AddNd(aclTensor *x1, aclTensor *x2, aclOpExecutor *executor)

L0 APIs reside in namespace l0op, and are named in ${op_type}${format}${dtype} format, where ${op_type} indicates the operator name, ${format} indicates the input/output data format of the operator, and ${dtype} indicates the input/output data type of the operator. (For unconventional input/output data types, the data type mapping must be specified.) The following is an example:

l0op::AddNd                               // The input of the Add operator is computed in ND format.
l0op::MatMulNdFp162Fp32                   // The input and output of the MatMul operator are computed in ND format. The value 2 indicates To, indicating that the input is fp16 and the output is fp32.
l0op::MatMulNzFp162Fp16                   // The input and output of the MatMul operator are computed in NZ format. The value 2 indicates To, indicating that both the input and output are fp16.

Level-2 APIs

L2 APIs, on the host, are higher-level encapsulation of L0 APIs (they call one or more L0 APIs to implement more flexible functions). This type of APIs allows for single-operator calls. You do not need to know how operators work inside, but just call the L2 APIs to use the operators.

The return value of L2 APIs is of the aclnnStatus type, which generally involves the two-phase APIs for obtaining workspaceSize and executing operators.

aclnnStatus aclnnXxxGetWorkspaceSize(const aclTensor *src, ..., aclTensor *out, ..., uint64_t *workspaceSize, aclOpExecutor **executor);
aclnnStatus aclnnXxx(void *workspace, uint64_t workspaceSize, aclOpExecutor *executor, aclrtStream stream);

The last two parameters of aclnnXxxGetWorkspaceSize are fixed at (uint64_t *workspaceSize, aclOpExecutor **executor). Their name and type cannot be changed.
The aclnnXxx API parameter is fixed at (void *workspace, uint64_t workspaceSize, aclOpExecutor *executor, aclrtStream stream).

aclnnXxxGetWorkspaceSize is the first-phase API that calculates the required workspace memory. After you get this size, allocate NPU memory accordingly, then call aclnnXxx to execute the computation. Xxx stands for the operator type, like Add.

The workspace refers to the temporary memory required by API computation on the Ascend AI Processor except for input and output.
The second-phase API aclnnXxx(...) cannot be called repeatedly. The following calls throw an exception:
1 2 3
aclnnXxxGetWorkspaceSize(...) aclnnXxx(...) aclnnXxx(...)

In-place Operator APIs
These APIs perform update operations directly at the original memory address. During computation, the input and output share the same address, minimizing redundant memory usage. In the aclnn class, the in-place operator APIs are typically named aclnnInplaceXxxGetWorkspaceSize (first-phase API) and aclnnInplaceXxx (second-phase API).

Parent topic: nnopbase APIs