Ascend C programming involves developing code on two different platforms (host and device). This chapter briefly describes the differences between the host and device to help you understand the heterogeneous system. It also describes the operator data flow, to help you further understand how to properly arrange the executions of operator code in the heterogeneous architecture.
Differences Between the Host CPU and Device NPU
- Different hardware resources
The CPU is designed to execute general-purpose computing tasks, but is not efficient in processing a large number of parallel computing tasks (such as matrix multiplication and batch processing). By contrast, the NPU aims to accelerate machine learning and deep learning tasks. It is good at performing a huge number of parallel computing tasks. The NPU contains a lot of dedicated hardware. Specifically, the cube unit supports matrix computation, with a single core processing a 16 × 16 × 16 multiply–accumulate operation of the float16 type within one clock cycle. The Vector unit supports vector computation, with a single core processing a 128 + 128 addition of the float16 type within one clock cycle.
- Different physical memory spaces
The physical memories of the host and device are separate, and data usually needs to be exchanged between them.
How to Properly Arrange Operator Code
The host and device can be deemed as a collaborative heterogeneous system for allocating the work that they are good at to each processing unit. Non-computing-intensive tasks (usually scalar tasks) are recommended on the host. Computing-intensive tasks are recommended on the device. The Single Instruction/Multiple Data (SIMD) of the device NPU can be used to efficiently implement matrix and vector operations for batch data.
Ascend C operator implementation consists of two parts:
- Tiling implementation on the host
As internal storage of the AI Core on the NPU cannot store all the input and output data of operators, the input data is tiled into different parts. The first part is transferred in, computed, and then transferred out, so does the next part. This process is called tiling. The algorithm for splitting data is called the tiling algorithm or tiling strategy. Then, a computation program, called tiling implementation or tiling function, determines tiling parameters (such as the block size transferred each time and the total number of cycles) based on operator information such as shape. The AI Core is not good at scalar computation in the tiling implementation. Therefore, this computation is executed on the host CPU independently.
- Kernel implementation on the device
Kernel implementation refers to the implementation of operator kernel functions. In kernel functions, the tiling structure transferred from the host is parsed to obtain the tiling information, which is used to control the process of transferring data in and out of the local memory. The operator logic is implemented by calling the computing, data movement, memory management, and task synchronization APIs. The core logic is that these are mostly computing-intensive tasks and are better executed on the device NPU.
Operator Data Flow
Data exchanges between the host and device happen during operator execution. The following provides the specific data flow for the passing of tiling parameters. First, the tiling algorithm on the host computes tiling parameters based on the operator input and output and stores the parameters in the tiling structure. Then, the tiling structure on the host is sent to the device. Finally, the operator on the device obtains and parses the tiling structure, and executes the subsequent operator compute logic based on the information.
