Working Principle

Operator Delivery Process

Output Shape Inference (inferShape)

Shape inference (inferShape) is used to infer the output shape and data type based on the input shape and attributes.

For example, if the left matrix tensorA of matrix multiplication Matmul is M × K, the right matrix tensorB is K × N, and the result of inferShape is M × N, as shown in the following figure.

If the input data type is float16, the output data type is float16 by default.

Figure 1 Shape inference example

Tiling

In most cases, the storage space of a single AI core is limited, and the input data of an operator cannot be completely loaded at a time for computation. The input needs to be divided into multiple blocks and computed by block. This process is called tiling, and the data division algorithm is called the tiling algorithm or tiling policy.

The tiling policy has a great impact on the performance of complex operators. The performance of the same operator may vary by 10 times under different tiling policies.

Figure 2 Tiling example

Obtaining the Workspace Size

Workspace: Additional device memory sometimes is required for data exchange or caching in the operator. The workspace needs to be allocated before the operator is executed. Generally, there are two usage scenarios:

1. If the space on the unified buffer and L1 buffer is insufficient, you can temporarily store data in the workspace.

2. When calling other APIs, you can use the workspace as the input parameter.

Generally, the tiling and workspace are calculated together.

Figure 3 Workspace example

Operator Delivery LaunchKernel

Figure 4 Operator delivery diagram

Operator Execution

Figure 5 Operator execution diagram