[object Object]

Note: This API will be deprecated in later versions. Use the latest aclnnGroupedMatmulV5 API.

[object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication, supporting non-uniform matrix dimension sizes across multiple groups. The basic function is matrix multiplication, for example, $y_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups, and $m_i$, $k_i$, and $n_i$ indicate the corresponding dimension sizes. Both input and output parameters are of the aclTensorList type, with the following functions:

  • K-axis grouping: $k_i$ varies across groups, while $m_i$ and $n_i$ remain the same for each group. In this case, $x_i$ and $weight_i$ can be concatenated along the K-axis.
  • M-axis grouping: $k_i$ remains the same for each group. In this case, $weight_i$ and $y_i$ can be concatenated along the N-axis.

  Compared with , this API has the following new features:

  • The values in [object Object] can be the sizes of groups along the grouping axis.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
    • Static quantization (per-tensor and per-channel), BFLOAT16 and FLOAT16 outputs, with or without activation (For details, refer to . Same below.)
    • Dynamic quantization (per-token + per-channel), BFLOAT16 and FLOAT16 outputs, with or without activation
    • Fake-quantization with INT4 input [object Object] without activation in per-channel and per-group modes
  • Ascend 950PR/Ascend 950DT:
    • Static quantization (1. pertensor-perchannel (T-C); 2. pertensor-pertensor (T-T)) with BFLOAT16, FLOAT16, and FLOAT32 outputs, with bias.
    • Dynamic quantization (1. pertoken-perchannel (K-C); 2. pertoken-pertensor (K-T); 3. pertensor-pertensor (T-T); 4. pertensor-perchannel (T-C); 5. mx quantization; 6. pergroup-perblock (G-B)) with BFLOAT16, FLOAT16, and FLOAT32 outputs, with bias.
    • Fake-quantization with INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, or HIFLOAT8 input [object Object] without activation in per-channel mode only.

Notes:

• "Single-tensor" means that tensors of all groups in a tensor list are concatenated into one tensor along the axis specified [object Object].

• Tensor transpose: If the tensor shape is [M, K], the stride is [1, M], and the data layout is [K, M], then the tensor is a non-contiguous tensor.

• Formula:

  • Non-quantization scenario:

    $$y_i=x_i\times weight_i + bias_i$$

  • Quantization scenario (static quantization, T-C && T-T, without perTokenScaleOptional):

    $$y_i=(x_i\times weight_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32, without offset

      $$y_i=(x_i\times weight_i) * scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, T-T && T-C && K-T && K-C):

    $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32

      $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, MX && G-B):

    $$y_i[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (xSlice_i * weightSlice_i)) * (per\_token\_scale_i[m/gsM, j] * scale_i[j, n/gsN])) + bias_i[n]$$

    $gsM$, $gsN$, and $gsK$ represent the quantization block sizes for the M, N, and K axes respectively. $xSlice_i$ denotes a vector of length $gsK$ from the $m$-th row of $x_i$, and $weightSlice_i$ denotes a vector of length $gsK$ from the $n$-th column of $weight_i$. Both tensors are sliced along the K-axis starting from $j \times gsK$, where $j \in [0, kLoops)$ and $kLoops = \lceil K_i / gsK \rceil$. Additionally, a final slice length less than $gsK$ is supported.

  • Fake-quantization scenario:

    $$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
    • x: FLOAT16, BFLOAT16, FLOAT32, INT8, and INT4
    • weight: FLOAT16, BFLOAT16, FLOAT32, INT8, and INT4; format: ND and FRACTAL_NZ
    • biasOptional: FLOAT16, FLOAT32, and INT32
    • scaleOptional: UINT64, BFLOAT16, and FLOAT32
    • perTokenScaleOptional: FLOAT32
    • out: FLOAT16, BFLOAT16, INT8, FLOAT32, and INT32
    • The input parameters x and weight, and the output parameter out support a maximum of 128 tensors.
  • Ascend 950PR/Ascend 950DT:
    • x supports FLOAT8_E4M3FN, FLOAT8_E5M2, INT8, HIFLOAT8, FLOAT16, BFLOAT16, FLOAT32 and FLOAT4_E2M1.
    • weight supports FLOAT8_E4M3FN, FLOAT8_E5M2, INT8, INT4, HIFLOAT8, FLOAT16, BFLOAT16, FLOAT32 and FLOAT4_E2M1. Only the ND format is supported.
    • biasOptional supports INT32, BFLOAT16, FLOAT16, and FLOAT32. When the input x is of type INT8, FLOAT16, BFLOAT16, or FLOAT32, biasOptional supports INT32, BFLOAT16, FLOAT16, and FLOAT32. When the input x is of type FLOAT4_E2M1, biasOptional supports only FLOAT32. For other types of inputs, a null pointer needs to be passed.
    • scaleOptional supports UINT64, INT64, BFLOAT16, FLOAT32, and FLOAT8_E8M0.
    • perTokenScaleOptional supports FLOAT32 and FLOAT8_E8M0.
    • groupListType cannot be set to 2.
    • actType supports 0, 1, 2, 4, and 5. For details about the constraints, see [object Object]Constraints[object Object].
    • out supports BFLOAT16, FLOAT16, and FLOAT32.
    • offsetOptional is not supported.
    • groupType supports grouping by m axis. Only non-quantization and quantization support grouping by k axis. Only non-quantization and fake-quantization support no grouping.
    • In the non-quantization scenario, the input parameters x and weight and the output parameter out support a maximum of 1024 tensors. In the fake-quantization scenario, the input parameters x and weight and the output parameter out support a maximum of 128 tensors. In the full quantization scenario, only a single tensor is supported.

• Return

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return

  [object Object]: status code. For details, see .

[object Object]

[object Object][object Object]

• Based on the precision processing of the input data ([object Object] and [object Object]) and the output matrix ([object Object]) during computation, the GroupedMatmul operator supports three primary scenarios: non-quantization, fake quantization, and full quantization.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    [object Object]undefined

  • Ascend 950PR/Ascend 950DT:

    For details, see .

[object Object][object Object]

• Formula [object Object][object Object]

  • Non-quantization scenario:$$y_i=x_i\times weight_i + bias_i$$

  [object Object][object Object]

  • Full-quantization scenario (without perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32$$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$

  • Full-quantization scenario (with perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16

      $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

    • [object Object] in INT4, no [object Object]

      $$y_i=x_i\times (weight_i * scale_i) * per\_token\_scale_i$$

  [object Object][object Object]

  • Fake-quantization scenario:

    • [object Object] in FLOAT16 or BFLOAT16 and [object Object] in INT4 or INT8 (supported only when [object Object], [object Object], and [object Object] are all single-tensor)$$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

    [object Object][object Object]

    • [object Object] in INT8 and [object Object] in INT4 (supported only when [object Object], [object Object], and [object Object] are all single-tensor) ([object Object] is a required parameter. It is an auxiliary result of offline computation. It is defined as $bias_i=8\times weight_i * scale_i$, reduced along the K-axis.)$$y_i=((x_i - 8) \times weight_i * scale_i+bias_i ) * per\_token\_scale_i$$

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.

[object Object]

• Common constraints

[object Object][object Object] - If [object Object] and [object Object] need to be transposed, the corresponding tensors must be . - The size of the last dimension for each tensor in [object Object] and [object Object] should be less than 65536. The last dimension of $x_i$ refers to the K-axis when [object Object] is not transposed or the M-axis when [object Object] is transposed. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. - When the [object Object] is FRACTAL_NZ, the shape of [object Object] must meet the requirements of the FRACTAL_NZ format. - Generally, [object Object] supports only 1D, and the length must match the M-axis size of [object Object]. This parameter only supports scenarios where [object Object], [object Object], and [object Object] are all single-tensor (with a TensorList length of 1). - When the length of the TensorList in the output is 1, [object Object] constrains the valid portion of the output data. Any portion not specified in [object Object] will not be updated. - When [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence, representing the cumulative sum (cumsum) results of the grouping axis sizes. When [object Object] is [object Object], it must be a non-negative sequence representing the size of each group along the grouping axis. When [object Object] is [object Object], it must be a non-negative sequence with a shape of [E, 2], where $E$ represents the group size. The data layout is [object Object], where [object Object] indicating the size of each group along the grouping axis. For details, see . - groupType indicates the axis to be grouped. For example, in matrix multiplication C[m,n] = A[m,k] x B[k,n], groupType can be set to -1 (no grouping), 0 (grouping by the M axis), or 2 (grouping by the K axis). For details, see the constraints in [object Object]Supported Scenarios for groupType[object Object]. - actType (int64_t, input for computation): integer, indicating the activation function type. The value ranges from 0 to 5.

[object Object]

[object Object][object Object]

[object Object]

• Common constraints:

  • [object Object]: The value can be 0 or 1. When groupListType is set to 0, groupListOptional must be a non-negative monotonic non-decreasing sequence. When groupListType is set to 1, groupListOptional must be a non-negative sequence.
  • actType (int64_t, input for computation): integer, indicating the activation function type. The value ranges from 0 to 5.
    • In fake-quantization and non-quantization scenarios, only 0 is supported for actType.
    • In full quantization scenarios, when x and weight are of the INT8 type, the quantization mode is static T-C quantization or dynamic K-C quantization, and the scale data type is FLOAT32 or BFLOAT16, the value of actType can be 0, 1, 2, 4, or 5. In other full quantization scenarios, only 0 is supported for actType.

  [object Object][object Object]

  [object Object]

  • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, and activationInputOptional.

  • The combinations of data types supported by the parameters that are not empty must meet the requirements in the following table.

    [object Object]undefined

  • The scaleOptional must meet the requirements in the following table (g indicates the number of matmul groups, that is, the number of groups).

    [object Object]undefined
  [object Object][object Object]
  • The supported input types in the dynamic quantization (T-T && T-C && K-T && K-C quantization) scenario are as follows:

    • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

    • The data type combinations supported by the parameters that are not empty must meet the requirements in the following table:

      [object Object]undefined

    • The scaleOptional must meet the requirements listed in the following table (g indicates the number of matmul groups, that is, the number of groups). It is recommended that the shape of scaleOptional be (g,) in the pertensor scenario to avoid confusion with the G-B quantization mode.

      [object Object]undefined

    • perTokenScaleOptional must meet the following requirements:

      [object Object]undefined
  [object Object][object Object]

  • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

  • In the calculation formula, the quantization block size is as follows: gsM = gsN = 1, gsK = 32. mx quantization is a special pergroup quantization.

  • The combinations of data types supported by the parameters that are not empty must meet the requirements listed in the following table.

    [object Object]undefined

  • The scaleOptional must meet the requirements listed in the following table. Here, g indicates the number of matmul groups, and g_i indicates the ith group (starting from 0).

    [object Object]undefined

  • perTokenScaleOptional must meet the following requirements:

    [object Object]undefined

  • If the input x in the FLOAT4_E2M1 quantization is x, K must be an even number and cannot be 2. When the weight is not transposed, N must be an even number.

    [object Object]
  [object Object]

  • The following data types are supported in the dynamic quantization (G-B quantization) scenario:

  • The following input parameters are empty: biasOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, and activationInputOptional.

  • The quantization block size is calculated as follows: Currently, only gsM = 1 and gsN = gsK = 128 are supported.

  • The data type combinations supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

  • The scaleOptional parameter must meet the requirements listed in the following table. (g indicates the number of matmul groups, that is, the number of groups, and g_i indicates the ith group (the subscript starts from 0).)

    [object Object]undefined

  • The perTokenScaleOptional parameter must meet the following requirements:

    [object Object]undefined

  • Special processing in dynamic quantization scenarios:

    • In the dynamic quantization scenario where M or K is grouped, if N is 1 and the shape of scaleOptional is (g, 1), and the weight can be quantized in either per-tensor or per-channel mode, the per-tensor quantization mode is preferred.
    • In the dynamic quantization scenario where M is grouped, if g = M and the shape of perTokenScaleOptional is (g,), the pertoken quantization mode is selected for x. If g = M, K <= 128, and the shape of perTokenScaleOptional is (g, 1), the quantization mode of x is selected based on the quantization mode of the weight. If the weight is quantized in per-channel or per-tensor mode, the pertensor quantization mode is selected for x. If the weight is quantized in per-block mode, the pergroup quantization mode is selected for x.
    • In the dynamic quantization scenario where K is grouped, if K is less than 128, N is less than or equal to 128, and the shape of scaleOptional is (g, 1), the quantization mode can be either non-pergroup or G-B quantization according to the existing quantization mode differentiation rules. In this scenario, the G-B quantization mode is used.
    • In the dynamic quantization scenario with K groups, when M is equal to 1 and the shape of perTokenScaleOptional is (g, 1), if x can be either pertoken or pertensor quantized, the pertensor quantization mode is preferred.
    • In the dynamic quantization scenario with K groups, if K is less than 128, M is equal to 1, and the shape of perTokenScaleOptional is (g, 1), then: If N is less than or equal to 128, pergroup quantization is selected for x. If N is greater than 128, the quantization mode of x is selected based on the quantization mode of the weight. If the weight is perchannel or pertensor quantization, pertensor quantization is selected for x. If the weight is perblock quantization, pergroup quantization is selected for x.
    • In the dynamic quantization scenario with K groups, if K is less than 128 and M is not equal to 1, then: If N is less than or equal to 128, pergroup quantization is selected for x. If N is greater than 128, the quantization mode of x is selected based on the quantization mode of the weight. If the weight is perchannel or pertensor quantization, perToken quantization is selected for x. If the weight is perblock quantization, pergroup quantization is selected for x.[object Object]
  [object Object]
  • The following data types are supported in non-quantization scenarios:

    • The following input parameters are empty: scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

    • The data type combinations supported by the parameters that are not empty must meet the requirements in the following table.

      [object Object]undefined
  [object Object][object Object]
  • The fake-quantization scenario supports the following data types:

    • The following input parameters are empty: scaleOptional, offsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional and activationQuantOffsetOptional.

    • The combinations of data types supported by non-empty parameters must meet the requirements in the following table.

      [object Object]undefined

    • When the data type of weight is FLOAT8_E5M2, FLOAT8_E4M3FN or HIFLOAT8, antiquantOffsetOptional can only be set to a null pointer or null tensor list, and weight can only be transposed.

    • If the weight type is INT4, the last dimension of each group of tensors in weight must be an even number. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed.

    • The following table describes the requirements for antiquantScaleOptional and non-empty biasOptional and antiquantOffsetOptional (g indicates the number of matmul groups, that is, the number of groups).

      [object Object]undefined
  [object Object][object Object]
  • Supported scenarios for different [object Object] values:

    • "S" stands for single-tensor, and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

      [object Object]undefined
  [object Object]

[object Object][object Object]

The following example is for reference only. For details, see .