Channel Split of Matrix Multiplication Outputs

Overview

Channel split of matrix multiplication outputs, or ChannelSplit, refers to matrix C, the Matmul computation result, being stored in a fractal format when the format of matrix C is NZ. For details about the NZ format, see the data formats. When the physical layout format of matrix C is NZ and the data type is float, each fractal contains 16 × 16 elements by default, meaning that the size of each fractal is 16 × 16. The ChannelSplit function divides each 16 × 16 fractal of matrix C into 16 × 8 fractals, allowing matrix C to be stored in 16 × 8 fractals.

Since the size of one float data element is 4 bytes, the 16 × 8 fractals are 32-byte aligned on the inner axis. The amount of data on the inner axis is consistent with the data units processed by an NPU vector computing instruction, facilitating subsequent computation. The ChannelSplit function is disabled by default. To enable it, you need to set the isEnableChannelSplit parameter in MatmulConfig.

Figure 1 ChannelSplit function

Application Scenarios

This function is used when matrix C of the NZ format and float type needs to be stored in 16 × 8 fractal format.

Restrictions

To enable the ChannelSplit function, the following conditions must be met:

• The data layout format of matrix C is CubeFormat::NZ.
• The data type of matrix C is float.
• The logical memory location of matrix C is the global memory.

Example

For a complete operator example, see matmul_channelsplit operator sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// Specify the MatmulConfig template to be obtained and modified.
constexpr static MatmulConfigMode configMode = MatmulConfigMode::CONFIG_NORM;
// Set the template parameter isEnableChannelSplit to true to enable the ChannelSplit function of the MatmulConfig template.
constexpr static MatmulFuncParams funcParamsChannelSplit{
    false, false, false, false, 0, IterateOrder::ORDER_M, ScheduleType::INNER_PRODUCT, true, false, false, false, true/*isEnableChannelSplit*/
};
constexpr static MatmulConfig MM_CFG = GetMMConfig<configMode>(funcParamsChannelSplit);
Matmul<A_TYPE, B_TYPE, C_TYPE, BIAS_TYPE, MM_CFG> mm;

// Perform the regular Matmul computation and output fractals of 16 × 8.
REGIST_MATMUL_OBJ(&pipe, GetSysWorkSpacePtr(), mm);
mm.SetTensorA(gm_a);
mm.SetTensorB(gm_b);
mm.SetBias(gm_bias);
mm.IterateAll(gm_c);