Documentation
Download

TCubeTiling Structure

The TCubeTiling structure contains parameters related to the Matmul tiling algorithm that are passed to the Matmul kernel for Matmul tiling, data movement, and computation. For details about the parameters of the TCubeTiling structure, see Table 1.

Table 1 Description for the TCubeTiling structure

Parameter

Data Type

Description

usedCoreNum

int

Number of AI Processor cores used. Set this parameter based on your actual requirements. Value range: [1, Maximum number of AI Processor cores]. The relationship between this parameter and shape-related parameters is as follows: usedCoreNum = (M/singleCoreM) × (N/singleCoreN).

M, N, Ka, Kb

int

Shape size of the original input of matrices A, B, and C, in elements. M and Ka are the original input shapes of matrix A, and Kb and N are the original input shapes of matrix B.

  • Size constraints
    In the following scenarios, intrinsicsCheck in MatmulConfig needs to be enabled to complete Matmul computation.

    • Matrix A in ND format:

      Without transpose, intrinsicsCheck must be enabled and M has no size limit if Ka is greater than 65535.

      With transpose, intrinsicsCheck must be enabled and Ka has no size limit if M is greater than 65535.

    • Matrix B in ND format:

      Without transpose, intrinsicsCheck must be enabled and Kb has no size limit if N is greater than 65535.

      With transpose, intrinsicsCheck must be enabled and N has no size limit if Kb is greater than 65535.

  • Alignment constraints

    • If the input of matrix A is in NZ format, M must be 16-element aligned and Ka must be aligned with C0_size.

      If the input of matrix B is in NZ format, Kb must be aligned with C0_size and N must be 16-element aligned.

    • If matrices A and B are in ND format, there is no alignment constraint.

    Notes:

    For inputs in NZ format, the value of C0_size depends on the data type:

    For float, the value is 8.

    For half or bfloat16_t, the value is 16.

    For int8_t, the value is 32.

    For int4b_t, the value is 64.

singleCoreM, singleCoreN, singleCoreK

int

Shape sizes of matrices A, B, and C in a single core, in elements. The value of this parameter must be greater than 0.

singleCoreK = K (K is not tiled during multi-core processing); singleCoreM <= M; singleCoreN <= N

Notes:

If the input of matrix A is in NZ format, singleCoreM must be 16-element aligned and singleCoreK must be aligned with C0_size × fractal_num.

If the input of matrix B is in NZ format, singleCoreK must be aligned with C0_size × fractal_num and singleCoreN must be 16-element aligned.

For inputs in NZ format, the values of C0_size and fractal_num depend on the data type:

For half or bfloat16_t, C0_size is 16 and fractal_num is 1.

For float, C0_size is 8 and fractal_num is 2.

For int8_t, C0_size is 32 and fractal_num is 1.

For int4b_t, C0_size is 64 and fractal_num is 1.

fractal_num represents the number of C0_size units required to meet alignment requirements during computation.

baseM, baseN, baseK

int

Shape sizes of matrices A, B, and C involved in a matrix multiplication instruction, in elements.

The shape size of matrices A, B, and C participating in a matrix multiplication must be fractal-aligned. For details, see the data format description in Mmad.

Note: The value of this parameter must be greater than 0.

depthA1, depthB1

int

Number of full-load base blocks in A1 and B1. depthA1 indicates the number of full-load baseM × baseK blocks in A1, and depthB1 indicates the number of full-load baseN × baseK blocks in B1.

Note: The value of this parameter must be greater than 0.

stepM, stepN, stepKa, stepKb,

int

stepM is a multiple of baseM of the left matrix in the bufferM direction buffered in A1.

stepN is a multiple of baseN of the right matrix in the buffer N direction buffered in B1.

stepKa is a multiple of baseK of the left matrix in the buffer Ka direction buffered in A1.

stepKb is a multiple of baseK of the right matrix in the buffer Kb direction buffered in B1.

Note: The value of this parameter must be greater than 0.

isBias

int

Whether to enable the bias. Values:

  • 0 (default): disables the bias.
  • 1: enables the bias.

Note: This parameter cannot be set to any other values. Otherwise, the parameter behavior is undefined.

transLength

int

max(A1Length, B1Length, C1Length, BiasLength). A1Length, B1Length, C1Length, and BiasLength indicate the sizes of UB space that needs to be temporarily occupied by the A, B, C, and Bias matrices during computation, respectively.

iterateOrder

int

Each Iterate call computes a slice of matrix C of the [baseM, baseN] size. After Iterate is complete, Matmul automatically offsets the matrix C position output by Iterate next time. iterOrder indicates the automatic offset order. Values:

  • 0: offsets along the M-axis direction first and then along the N-axis direction.
  • 1: offsets along the N-axis direction first and then along the M-axis direction.

Note: This parameter cannot be set to any other values. Otherwise, the parameter behavior is undefined.

dbL0A, dbL0B,

dbL0C

int

Whether to enable double buffering for MTE1.

dbL0A: Whether to enable double buffering for the left matrix MTE1. dbL0B: Whether to enable double buffering for the right matrix MTE1. dbL0C: Whether to enable double buffering for MMAD. Values:

  • 1: disables double buffering.
  • 2: enables double buffering.

Note: This parameter cannot be set to any other values. Otherwise, the parameter behavior is undefined.

shareMode

int

This parameter is reserved and can be ignored.

shareL1Size

int

This parameter is reserved and can be ignored.

shareL0CSize

int

This parameter is reserved and can be ignored.

shareUbSize

int

This parameter is reserved and can be ignored.

batchM

int

This parameter is reserved and can be ignored.

batchN

int

This parameter is reserved and can be ignored.

singleBatchM

int

This parameter is reserved and can be ignored.

singleBatchN

int

This parameter is reserved and can be ignored.

In most cases, you can call the GetTiling API to obtain the TCubeTiling structure. For details, see Matmul Tiling Usage Instructions. If you customize the TCubeTiling parameters, the value of each parameter must meet the requirements described in Table 1 and Table 2. If you need to modify the tiling after obtaining the TCubeTiling structure by calling the GetTiling API, set the parameters according to the following TCubeTiling parameter constraints and recommended values for performance tuning.

  • TCubeTiling constraints
    A group of valid TCubeTiling parameters must meet all the constraints listed in Table 2. If the MatmulConfig template of a Matmul object is an MDL template, the constraints listed in Table 2 must also be met in addition to those listed in Table 3.
    Table 2 TCubeTiling constraints

    Constraint

    Description

    usedCoreNum <= aiCoreCnt

    The number of used cores is less than or equal to the maximum number of cores configured in the current AI processor.

    baseM * baseK * sizeof(A_type) * dbL0A< l0a_size

    The size of the base block of matrix A does not exceed the size of the l0a buffer.

    baseN * baseK * sizeof(B_type) * dbL0B < l0b_size

    The size of the base block of matrix B does not exceed the size of the l0b buffer.

    baseM × baseN × sizeof(l0c_type) × dbL0C < l0c_size, where l0c_type is of the int32_t or float data type.

    The size of a base block of matrix C does not exceed the size of the l0c buffer.

    baseN * sizeof(Bias_type) < biasT_size

    The size of the base block of Bias does not exceed the size of the BiasTable buffer.

    stepM × stepKa × db = depthA1

    db indicates whether double buffering is enabled for the left matrix MTE2, that is, whether double buffering is enabled for L1. The value can be 1 (double buffering disabled) or 2 (double buffering enabled).

    The value of depthA1 is the same as that of stepM × stepKa × db.

    stepN × stepKb × db = depthB1

    db indicates whether double buffering is enabled for the right matrix MTE2, that is, whether double buffering is enabled for L1. The value can be 1 (double buffering disabled) or 2 (double buffering enabled).

    The value of depthB1 is the same as that of stepN × stepKb × db.

    For caching blocks in L1 memory, the block size of matrix A (AL1Size) and matrix B (BL1Size) must meet the following conditions:

    • Non-bias scenario

      AL1Size + BL1Size <= L1_size

    • Bias scenario

      AL1Size + BL1Size + baseN * sizeof(Bias_type) <= L1_size

    AL1Size and BL1Size are calculated as follows:
    • Transpose scenario:

      AL1Size = CeilDiv(baseM, C0_size) * baseK * depthA1 * sizeof(A_type)

      BL1Size = baseN * baseK * depthB1 * sizeof(B_type)

    • Non-transpose scenario:

      AL1Size = baseM * baseK * depthA1 * sizeof(A_type)

      BL1Size = CeilDiv(baseN, C0_size)* baseK * depthB1 * sizeof(B_type)

    Matrix A, matrix B, and bias meet the buffer size limit of the L1 Buffer block.

    Notes: The value of C0_size depends on the data type.

    For float, the value is 8.

    For half or bfloat16_t, the value is 16.

    For int8_t, the value is 32.

    For int4b_t, the value is 64.

    baseM × baseK, baseK × baseN and baseM × baseN are fractal-aligned in NZ format.

    The base blocks of matrix A, matrix B, and matrix C must meet the following alignment constraints:

    • baseM and baseN must be aligned to 16 elements. If matrix A is not transposed and matrix B is transposed, baseK must be aligned to C0_size. In other scenarios (matrix A is transposed or matrix B is not transposed), baseK must be aligned to 16 elements.

    Notes: The value of C0_size depends on the data type.

    For float or int32_t, the value is 8.

    For half or bfloat16_t, the value is 16.

    For int8_t, the value is 32.

    For int4b_t, the value is 64.
    Table 3 Additional constraints for the MDL template

    Constraint

    Description

    When data in the Ka direction is not fully loaded, that is, Ka/baseK > stepKa, stepM = 1.

    When data in the K direction is not fully loaded, data in the M direction must be moved block-wise.

    When data in the Kb direction is not fully loaded, that is, Kb/baseK > stepKb, stepN = 1

    When data in the K direction is not fully loaded, data in the N direction must be moved block-wise.

    kaStepIter_ % kbStepIter_ = 0 or kbStepIter_ % kaStepIter_ = 0

    kaStepIter_ = CeilDiv(tiling_->singleCoreK_, tiling_->baseK * tiling_->stepKa)

    kbStepIter_ = CeilDiv(tiling_->singleCoreK_, tiling_->baseK * tiling_->stepKb)

    For the K-direction cyclic movement in the MDL template, the numbers of iterations in the Ka and Kb directions must be multiples of each other.

    kaStepIter_: number of cyclic movement iterations in the Ka direction

    kbStepIter_: number of cyclic movement iterations in the Kb direction
  • Recommended values for performance tuning

    Based on the tiling tuning experience, the recommended values or example values for some TCubeTiling parameters are as follows:

    • Recommended base block (baseM, baseN, baseK): (128, 256, 64)
    • dbL0A / dbL0B = 2
    • depthA1 / (stepM * stepKa) = 2
    • depthB1 / (stepN * stepKb) = 2
    • Set the stepKa and stepKb parameters first to ensure that the K direction is fully loaded, and then the M or N direction.
Parent topic: Matmul Tiling
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details