文档

CANN Operator Types

Figure 1 shows the location of CANN operators in the Ascend AI Processor.
  • The AI Core is the compute core of the Ascend AI Processor. It executes operator tasks that are intensive in cube, vector, and scalar computation. Operators executed on the AI Core are called Tensor Boost Engine (TBE) operators.
  • The AI CPU executes operators that are not suitable for running on the AI Core and is responsible for non-cube and selection-intensive computation with complex logic. The operators executed on the AI CPU are a supplement to the AI Core operators and are called AI CPU operators.
Figure 1 CANN operators in the Ascend AI Processor
There are two types of CANN operators: TBE operators and AI CPU operators.
  • TBE operators

    TBE operators run on the AI Core with strong compute power, which executes compute-intensive operators related to cubes, vectors, and scalars.

    TBE supports custom operator development based on the Tensor Virtual Machine (TVM) framework. TBE provides two operator development modes: Domain-Specific Language (DSL) and Tensor Iterator Kernel (TIK). You can select either development mode as required. The differences between them are as follows:

    • DSL mode

      DSL APIs are highly encapsulated. You simply need to use the APIs to express the computing process. Subsequent operator schedule, optimization, and building can be performed using existing APIs, which is beginner friendly.

    • TIK mode

      You can use the TIK API calls to build a custom operator in Python. Then, the TIK compiler compiles the operator into a binary file of an application that adapts to Ascend AI Processor. The TIK mode requires manual control in data movement and computation processes, which is difficult to get started, but it offers flexibility and better performance by fully utilizing hardware capabilities.

  • AI CPU operators

    AI CPU operators run on the AI CPU. You may wish to implement AI CPU custom operators in the following scenarios:

    • You need operators that are not suitable for running on the AI Core, including operators for non-cube complex computation and selection-intensive operators with complex logic.

      For example, control operators such as Dump and Profiling, resource status operators such as Queue and Stack, and search operators such as TopK and Where.

    • You need operators unsupported by the AI Core due to unsupported data types. For example, operators require complex32 and complex64, which are not supported by the AI Core.
    • You need to streamline the network execution process on Ascend AI Processor and it is difficult to implement TBE custom operators. You can customize AI CPU operators for efficient functionality commissioning. After functionality commissioning, convert the AI CPU custom operator into a TBE operator during performance commissioning.

If a TBE operator and an AI CPU operator of the same type both exist, the TBE operator will be executed first.

Parent topic: Background Knowledge