Basic Process

Tiling implementation on the host has been described in section "SIMD Operator Implementation". This section focuses on the programming mode and API usage when the CANN framework is connected.

In most cases, the local memory cannot completely store the operator input and output. Therefore, you need to move some inputs for computation multiple times until the complete final result is obtained. This process of data tiling and block computation is called tiling. Tiling implementation is a compute program that determines parameters related to the data tiling algorithm (such as the block size to be moved each time and the total number of cycles) based on information such as the shape of an operator.

After tiling is implemented, the obtained parameters related to the tiling algorithm are passed to the kernel to guide parallel data tiling. The AI Core is not good at scalar computation in the tiling implementation. Therefore, the AI Core is executed on the host CPU independently.

Figure 1 Input and output of tiling implementation

As shown in the preceding figure, tiling implementation is a process of determining parameters related to the tiling algorithm based on information such as the operator shape. The information such as the operator shape can be considered as the input of tiling implementation, and the parameters related to the tiling algorithm can be considered as the output of tiling implementation. The input and output are carried by the parameters (TilingContext* context structure) of the tiling function. You can obtain the input of tiling implementation: input, output, and attribute information of the operator from the context structure. After tiling, you can obtain the output of the tiling implementation: TilingData structure (parameters related to the tiling algorithm), numBlocks variable, TilingKey that is used to select different kernel implementation branches, and operator workspace, and then set the output to the context structure.

The concepts of TilingData, numBlocks, TilingKey, and workspace are described as follows:

Basic Process of Tiling

The following figure shows the development process of tiling.

Figure 2 Tiling development process

The following uses a simple Add operator as an example to describe the implementation process of tiling. In this sample, the shape size of the data to be processed can be evenly allocated to each core and aligned to the size of a data block (32 bytes).

Compile the header file for defining the TilingData structure of the operator. The file is named operator name_tiling.h and stored in the op_host directory of the operator project. The sample code is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
 
#ifndef ADD_CUSTOM_TILING_H
#define ADD_CUSTOM_TILING_H
#include "register/tilingdata_base.h"

namespace optiling {
BEGIN_TILING_DATA_DEF(TilingData)               // Register a tiling class and uses the tiling name as the input parameter.
  TILING_DATA_FIELD_DEF(uint32_t, totalLength); // Add the tiling field to compute the total data volume.
  TILING_DATA_FIELD_DEF(uint32_t, tileNum);     // Add the tiling field that specifies the total number of data blocks to be computed on each core.
END_TILING_DATA_DEF;
// Register the operator TilingData class with the corresponding AddCustom operator.
REGISTER_TILING_DATA_CLASS(AddCustom, TilingData)
}
#endif // ADD_CUSTOM_TILING_H

The procedure is as follows:

  1. Compile the code framework, add the judgment condition of #ifndef... to prevent repeated inclusion of header files, and include the register/tilingdata_base.h header file. Multiple macros for TilingData registration are defined in tilingdata_base.h. The sample code is as follows: 

    1
2
3
4
5
6
7
8
9
    		 
    #ifndef ADD_CUSTOM_TILING_H
#define ADD_CUSTOM_TILING_H
#include "register/tilingdata_base.h"

namespace optiling {
// Tiling structure definition and registration code
// ...
}
#endif // ADD_CUSTOM_TILING_H

  2. Design TilingData parameters, which are essentially related to parallel data tiling. In this example, the operator uses two tiling parameters: totalLength and tileNum. totalLength indicates the amount of data to be computed, and tileNum indicates the total number of computed data blocks on each core. For example, after totalLength is transferred to the kernel, the compute amount of each core can be obtained by dividing totalLength by the number of cores involved in the compute. In this way, multi-core data is tiled.
  3. Define the TilingData structure. Define a TilingData class by calling BEGIN_TILING_DATA_DEF, add the totalLength and tileNum fields by calling TILING_DATA_FIELD_DEF, and end the TilingData definition by calling END_TILING_DATA_DEF. For details about the APIs, see TilingData Structure Definition. 

    1
2
3
4
    		 
    BEGIN_TILING_DATA_DEF(TilingData)               // Register a tiling class and uses the tiling name as the input parameter.
  TILING_DATA_FIELD_DEF(uint32_t, totalLength); // Add the tiling field to compute the total data volume.
  TILING_DATA_FIELD_DEF(uint32_t, tileNum);     // Add the tiling field that specifies the total number of data blocks to be computed on each core.
END_TILING_DATA_DEF;

  4. Register the TilingData structure, call REGISTER_TILING_DATA_CLASS to register the TilingData class, and associate the TilingData class with the custom operator. The first parameter of REGISTER_TILING_DATA_CLASS is op_type (operator type). In this example, AddCustom is transferred, and the second parameter is the class name of TilingData. For details about the REGISTER_TILING_DATA_CLASS API, see TilingData Structure Registration. 

    1
2
    		 
    // Register the operator TilingData class with the corresponding AddCustom operator.
REGISTER_TILING_DATA_CLASS(AddCustom, TilingData)

Implement the tiling function in the .cpp file of the operator host. The file is named Operator name.cpp and stored in the op_host directory of the operator project. The prototype of the tiling function is definite. It accepts a TilingContext as the input. The input and output shape pointers can be obtained from the context. The registered tiling function is called by the framework, and the TilingContext parameter is passed during the call. The sample code is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
 
namespace optiling {
const uint32_t NUM_BLOCKS = 8;
const uint32_t TILE_NUM = 8;
static ge::graphStatus TilingFunc(gert::TilingContext *context)
{
    TilingData tiling;
    uint32_t totalLength = context->GetInputShape(0)->GetOriginShape().GetShapeSize();
    context->SetBlockDim(NUM_BLOCKS);
    tiling.set_totalLength(totalLength);
    tiling.set_tileNum(TILE_NUM);
    tiling.SaveToBuffer(context->GetRawTilingData()->GetData(), context->GetRawTilingData()->GetCapacity());
    context->GetRawTilingData()->SetDataSize(tiling.GetDataSize());
    size_t *currentWorkspace = context->GetWorkspaceSizes(1);
    currentWorkspace[0] = 0;
    return ge::GRAPH_SUCCESS;
}
} // namespace optiling

The procedure is as follows:

  1. Obtain TilingContext, that is, the input parameter gert::TilingContext* context of the tiling function.
  2. Set TilingData. After defining the TilingData class in step 3, you can create an instance of the class and set the value of each field by calling the set_{field_name} method (field_name is the tiling field name defined in step 3). After setting the tiling fields, call the SaveToBuffer method to serialize and save the TilingData instance.

    1. Obtain the input and output shape information based on the context. In this sample, the GetInputShape API of TilingContext is used to obtain the input shape size. 
      1
2
      		 
      // Obtain the input shape information.
uint32_t totalLength = context->GetInputShape(0)->GetOriginShape().GetShapeSize();
    2. Set TilingData. Call the set_{field_name} method to set the field value of TilingData. 
      1
2
3
4
5
      		 
      // Use TilingData to define a specific instance.
TilingData tiling;
// Set TilingData.
tiling.set_totalLength(totalLength);
tiling.set_tileNum(TILE_NUM);
    3. Call SaveToBuffer of the TilingData class to complete serialization and save the serialization to TilingContext. The first parameter of SaveToBuffer is the start address of the storage buffer, and the second parameter is the length of the buffer. Call GetRawTilingData to obtain the address of the typeless TilingData, and then call GetData to obtain the data pointer as the start address of the buffer. Call GetRawTilingData to obtain the address of the typeless TilingData, and then call GetCapacity to obtain the length of the TilingData as the length of the buffer. After the SaveToBuffer operation is complete, call SetDataSize to set the length of TilingData. The length is obtained by calling GetDataSize of the TilingData class. 
      1
2
3
      		 
      // Serialize and save the file.
tiling.SaveToBuffer(context->GetRawTilingData()->GetData(), context->GetRawTilingData()->GetCapacity());
context->GetRawTilingData()->SetDataSize(tiling.GetDataSize());

  3. Call SetBlockDim to set numBlocks. 

    1
    		 
    context->SetBlockDim(NUM_BLOCKS);

  4. (Optional) Call SetTilingKey to set TilingKey. 

    1
    		 
    context->SetTilingKey(1);

  5. (Optional) Call GetWorkspaceSizes to obtain the workspace size pointer and set the size. The following is only an example, where the size of workspace is set to 0: 

    1
2
    		 
    size_t *currentWorkspace = context->GetWorkspaceSizes(1);
currentWorkspace[0] = 0;

Parent topic: Tiling Implementation on the Host