[object Object]

[object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication. For example, $y_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups. Currently, M-axis grouping and K-axis grouping are supported. The corresponding functions are as follows:

  • M-axis grouping: $k_i$ and $n_i$ remain consistent for each group, while $m_i$ can vary.
  • K-axis grouping: $m_i$ and $n_i$ remain consistent for each group, while $k_i$ can vary.

• The basic computation formula is as follows (for details, see ):

  $$y_i=x_i\times weight_i + bias_i$$

• Version evolution:

  [object Object]undefined

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor containing the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • Ascend 950PR/Ascend 950DT:

    • The superscript "1" in the data type column of the preceding table indicates that the data type is not supported by this series.
    • The input parameters x and weight do not support the INT16 type, and x does not support the INT4 type.
    • In non-quantization scenarios, the input parameters x and weight and the output parameter out support a maximum of 1024 tensors. In fake-quantization scenarios, the input parameters x and weight and the output parameter out support a maximum of 128 tensors. In full-quantization scenarios, the input parameters x and weight and the output parameter out support a maximum of one tensor.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The superscript "2" in the "Data Type" column of the table above indicates data types that are not supported by the products.
    • FLOAT8_E5M2, FLOAT8_E4M3FN, HIFLOAT8, and FLOAT8_E8M0 are not supported.
    • The input parameter [object Object] does not support BFLOAT16.
    • The input parameter [object Object] does not support INT64.
    • The input parameters x and weight and the output parameter out support a maximum of 128 tensors.

• Return

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]undefined

• Return

  [object Object]: status code. For details, see .

[object Object]

[object Object][object Object]

• Based on the precision processing of the input data ([object Object] and [object Object]) and the output matrix ([object Object]) during computation, the GroupedMatmul operator supports three primary scenarios: non-quantization, fake quantization, and full quantization.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    [object Object]undefined

  • Ascend 950PR/Ascend 950DT:

    For details, see .

[object Object][object Object]

• Formulas [object Object][object Object]

  • Non-quantization scenario:$$y_i=x_i\times weight_i + bias_i$$

  [object Object][object Object]

  • Quantization scenario (static quantization, T-C && T-T, without perTokenScaleOptional):

    $$y_i=(x_i\times weight_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32, without offset

      $$y_i=(x_i\times weight_i) * scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, T-T && T-C && K-T && K-C):

    $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32

      $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, MX && G-B):

    $$y_i[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (xSlice_i * weightSlice_i)) * (per\_token\_scale_i[m/gsM, j] * scale_i[j, n/gsN])) + bias_i[n]$$

    $gsM$, $gsN$, and $gsK$ represent the quantization block sizes for the M, N, and K axes respectively. $xSlice_i$ denotes a vector of length $gsK$ from the $m$-th row of $x_i$, and $weightSlice_i$ denotes a vector of length $gsK$ from the $n$-th column of $weight_i$. Both tensors are sliced along the K-axis starting from $j \times gsK$, where $j \in [0, kLoops)$ and $kLoops = \lceil K_i / gsK \rceil$. Additionally, a final slice length less than $gsK$ is supported.

  [object Object][object Object]

  • Fake-quantization scenario:

    • [object Object] in FLOAT16 or BFLOAT16 and [object Object] in INT4 or INT8 (supported only when [object Object], [object Object], and [object Object] are all single-tensor)$$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

    [object Object][object Object]

    • [object Object] in INT8 and [object Object] in INT4 (supported only when [object Object], [object Object], and [object Object] are all single-tensor) ([object Object] is a required parameter. It is an auxiliary result of offline computation. It is defined as $bias_i=8\times weight_i * scale_i$, reduced along the K-axis.)$$y_i=((x_i - 8) \times weight_i * scale_i+bias_i ) * per\_token\_scale_i$$

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.

[object Object]

• Common constraints

[object Object][object Object] - If [object Object] and [object Object] need to be transposed, the corresponding tensors must be . - The size of the last dimension for each tensor in [object Object] and [object Object] should be less than 65536. The last dimension of $x_i$ refers to the K-axis when [object Object] is not transposed or the M-axis when [object Object] is transposed. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. - When the [object Object] is FRACTAL_NZ, the shape of [object Object] must meet the requirements of the FRACTAL_NZ format. - Generally, [object Object] supports only 1D, and the length must match the M-axis size of [object Object]. This parameter only supports scenarios where [object Object], [object Object], and [object Object] are all single-tensor (with a TensorList length of 1). - When the length of the TensorList in the output is 1, [object Object] constrains the valid portion of the output data. Any portion not specified in [object Object] will not be updated. - When [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence, representing the cumulative sum (cumsum) results of the grouping axis sizes. When [object Object] is [object Object], it must be a non-negative sequence representing the size of each group along the grouping axis. When [object Object] is [object Object], it must be a non-negative sequence with a shape of [E, 2], where $E$ represents the group size. The data layout is [object Object], where [object Object] indicating the size of each group along the grouping axis. For details, see . - groupType indicates the axis to be grouped. For example, in matrix multiplication C[m,n] = A[m,k] x B[k,n], groupType is set to -1 (no grouping), 0 (grouping by axis m), or 2 (grouping by axis k). For details, see the constraints on [object Object]Supported Scenarios for groupType[object Object]. - [object Object] (int64_t, computation input): integer type, indicating the activation function type. The value ranges from 0 to 5. The supported enumerated values are as follows: * 0: GMMActType::GMM_ACT_TYPE_NONE * 1: GMMActType::GMM_ACT_TYPE_RELU * 2: GMMActType::GMM_ACT_TYPE_GELU_TANH * 3: GMMActType::GMM_ACT_TYPE_GELU_ERR_FUNC (not supported) * 4: GMMActType::GMM_ACT_TYPE_FAST_GELU * 5: GMMActType::GMM_ACT_TYPE_SILU

[object Object]

[object Object][object Object]

[object Object]

• Common constraints:

  • groupType: Grouping by the m axis is supported, and the k axis can be grouped or not. Only non-quantization and full quantization support grouping by the k axis.
  • [object Object]: The value can be 0 or 1. When groupListType is set to 0, groupListOptional must be a non-negative monotonic non-decreasing sequence. When groupListType is set to 1, groupListOptional must be a non-negative sequence.
  • [object Object]: This parameter is not supported.
  • actType (int64_t, input for computation): integer parameter, indicating the activation function type. The value ranges from 0 to 5.
    • In fake-quantization and non-quantization scenarios, actType can only be set to 0.
    • In full quantization scenarios, when x and weight are of the INT8 type, the quantization mode is static T-C quantization or dynamic K-C quantization, and the scale data type is FLOAT32 or BFLOAT16, actType can be set to 0, 1, 2, 4, or 5. In other full quantization scenarios, actType can only be set to 0.

  [object Object][object Object]

  [object Object][object Object]undefined

  • The scaleOptional must meet the requirements listed in the following table (g indicates the number of matmul groups, that is, the number of groups):

    [object Object]undefined
  [object Object][object Object]
  • The supported input types in the T-T && T-C && K-T && K-C quantization scenario are as follows:

    • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

    • The combinations of data types supported by non-empty parameters must meet the requirements in the following table.

      [object Object]undefined

    • The scaleOptional parameter must meet the requirements in the following table (g indicates the number of matmul groups, that is, the number of groups). It is recommended that the shape of scaleOptional be set to (g,) in the per-tensor scenario to avoid confusion with the G-B quantization mode.

      [object Object]undefined

    • perTokenScaleOptional must meet the following requirements:

      [object Object]undefined
  [object Object][object Object]

  • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

  • In the calculation formula, the quantization block size is as follows: gsM = gsN = 1, gsK = 32. mxQuant is a special per-group quantization.

  • The following table lists the data type combinations supported by the parameters that are not empty.

    [object Object]undefined

  • The scaleOptional parameter must meet the requirements described in the following table. Here, g indicates the number of matmul groups, and g_i indicates the ith group (the subscript starts from 0).

    [object Object]undefined

  • perTokenScaleOptional must meet the following requirements:

    [object Object]undefined

  • When the input x of the mx quantization is FLOAT4_E2M1, K must be an even number and cannot be 2. If the weight is not transposed, N must be an even number.

    [object Object]
  [object Object]

  • The following data types are supported in the dynamic quantization (G-B quantization) scenario:

  • The following input parameters are empty: biasOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

  • The quantization block size is calculated as follows: Currently, only gsM = 1 and gsN = gsK = 128 are supported.

  • The following table lists the supported data type combinations for parameters that are not empty.

    [object Object]undefined

  • The scaleOptional must meet the requirements in the following table. (g indicates the number of matmul groups, and g_i indicates the ith group (the subscript starts from 0).)

    [object Object]undefined

  • The perTokenScaleOptional must meet the requirements in the following table.

    [object Object]undefined

  • Special processing in dynamic quantization scenarios:

    • In the dynamic quantization scenario where the M or K group is used, if N is equal to 1 and the shape of scaleOptional is (g, 1), and the weight can be quantized in both perTensor and perChannel modes, the perTensor quantization mode is preferred.
    • In the dynamic quantization scenario with M groups, when g = M and the shape of perTokenScaleOptional is (g,), x selects the per-token quantization mode. When g = M, K <= 128, and the shape of perTokenScaleOptional is (g, 1), the quantization mode of x is selected based on the quantization mode of the weight. If the weight is per-channel or per-tensor quantized, x is per-tensor quantized. If the weight is per-block quantized, x is per-group quantized.
    • In the dynamic quantization scenario with K groups, when K is less than 128, N is less than or equal to 128, and the shape of scaleOptional is (g, 1), the quantization mode can be either non-per-group quantization or G-B quantization according to the existing quantization mode differentiation rules. In this scenario, G-B quantization is used.
    • In the dynamic quantization scenario with K groups, when M is equal to 1 and the shape of perTokenScaleOptional is (g, 1), if x can be either per-token or per-tensor quantized, the per-tensor quantization mode is preferred.
    • In the dynamic quantization scenario with K groups, when K is less than 128, M is equal to 1, and the shape of perTokenScaleOptional is (g, 1), if N is less than or equal to 128, x is per-group quantized. If N is greater than 128, the quantization mode of x is selected based on the quantization mode of the weight. If the weight is per-channel or per-tensor quantized, x is per-tensor quantized. If the weight is per-block quantized, x is per-group quantized.
    • In the dynamic quantization scenario with K groups, when K is less than 128 and M is not equal to 1, if N is less than or equal to 128, x is per-group quantized. If N is greater than 128, the quantization mode of x is selected based on the quantization mode of the weight. If the weight is per-channel or per-tensor quantized, x is per-token quantized. If the weight is per-block quantized, x is per-group quantized.[object Object]

• In non-quantization scenarios, the following data types are supported:

  • The following input parameters are empty: scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

  • The combinations of data types supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

• The fake-quantization scenario supports the following data types:

  • The following input parameters are empty: scaleOptional, offsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional and activationQuantOffsetOptional.

  • The combinations of data types supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

  • When the data type of weight is FLOAT8_E5M2, FLOAT8_E4M3FN or HIFLOAT8, antiquantOffsetOptional can only be set to a null pointer or null tensor list, and weight can only be transposed.

  • If the data type of weight is INT4, the last dimension of each group of tensors in weight must be an even number. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed.

  • The following table describes the requirements for non-empty biasOptional, antiquantOffsetOptional, and antiquantScaleOptional. E indicates the number of matmul groups.

    [object Object]undefined
  [object Object]
  • Supported scenarios for different [object Object] values:

    • In the supported scenarios, single indicates a single tensor, and multiple indicates multiple tensors. The sequence is x, weight, and out. For example, single-multiple-single indicates that x is a single tensor, weight is multiple tensors, and out is a single tensor.

      [object Object]undefined
  [object Object]

[object Object][object Object]

The following example is for reference only. For details, see .