[object Object]

[object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication, supporting non-uniform matrix dimension sizes across multiple groups. The basic function is matrix multiplication, for example, $y_i[m_i,n_i]=x_i[m_i,k_i] times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups and $m_i$, $k_i$, and $n_i$ define the shapes for each group. Both inputs and outputs are of the aclTensorList type, with the following functions:

  • K-axis grouping: $k_i$ varies across groups, while $m_i$ and $n_i$ remain the same for each group. In this case, $x_i$ and $weight_i$ can be concatenated along the K-axis.
  • M-axis grouping: $k_i$ remains the same for each group. In this case, $weight_i$ and $y_i$ can be concatenated along the N-axis.

  New features compared with :

  • The data format of the input weight supports the AI processor affinity data layout format (FRACTAL_NZ).
  • The [object Object] parameter is added. It is an integer that indicates the quantization group size in per-group mode. If per-group quantization is not involved, set this parameter to [object Object].
  • Ascend 950PR/Ascend 950DT: The quantGroupSize parameter is not supported currently.

• Formulas:

  [object Object][object Object]

  • Non-quantization scenario:$$y_i=x_i \times weight_i + bias_i$$

  [object Object][object Object]

  • Quantization scenario (without perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i \times weight_i + bias_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32, without offset

      $$y_i=(x_i \times weight_i) * scale_i + bias_i$$

  • Quantization scenario (with perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i \times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32

      $$y_i=(x_i \times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

  • Quantization scenario (MX quantization, without bias or activation layer):

    $$y_i=(x_i * per\_token\_scale_i) \times (weight_i * scale_i)$$

  [object Object][object Object]

  • Dequantization scenario:$$y_i=(x_i \times weight_i + bias_i) * scale_i$$

  [object Object][object Object]

  • Fake-quantization (perchannel and pergroup) scenario:

    $$y_i=x_i \times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

  • Fake-quantization (mx) scenario:

    x is of type BFLOAT16/FLOAT16, and weight is of type FLOAT32 (indicating eight FLOAT4_E2M1 /FLOAT4_E2M1).

    $$y_i=x_i \times (weight_i * antiquant\_scale_i) + bias_i$$

    x is of type FLOAT8_E4M3FN, and weight is of type FLOAT32 (indicating eight FLOAT4_E2M1 /FLOAT4_E2M1).

    $$y_i=(x_i * per\_token\_scale_i) \times (weight_i * antiquant\_scale_i) + bias_i$$

  • Fake-quantization (K-CG) scenario:

    $$y_i=(x_i \times (weight_i * antiquant\_scale_i)) * scale_i * per\_token\_scale_i + bias_i$$

    In the preceding information, antiquant_scale_i is the per-group quantization parameter of the weight matrix, scale_i is the per-channel quantization parameter of the weight matrix, and per_token_scale_i is per-token quantization parameter.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
    • The superscript "1" in the "Data Type" column of the table above indicates data types that are supported by the products, and the superscript "2" indicates data types that are not supported by the products.
    • The [object Object] format can be converted from ND to NZ by calling [object Object] and [object Object]. When the input data type is INT32, each INT32 value is treated as eight INT4 values inside the API.
    • The input parameters x and weight, and the output parameter out support a maximum of 128 tensors.
  • Ascend 950PR/Ascend 950DT:
    • In the preceding table, the superscript 2 in the data type column indicates the data types supported by the series.
    • x supports FLOAT16, BFLOAT16, FLOAT8_E4M3FN and INT8.
    • weight supports FLOAT16, BFLOAT16, FLOAT4_E2M1, INT8 and INT4. The FRACTAL_NZ format is supported. If either of the last two axes is 1 (that is, n = 1 or k = 1), the proprietary format is not supported and this API cannot be called. You can use the aclnnNpuFormatCast API to convert the input format from ND to the format (NZ) that is more affinity to the AI processor. If the original weight is in the transposed state and you want to use the non-transposed path with higher performance for computation, you can use the aclnnPermute API to convert the weight to the non-transposed state and then call the aclnnNpuFormatCast API. When the data type is FLOAT4_E2M1, you also need to call the aclnnCast API to convert the FLOAT4_E2M1 represented by FLOAT32 to the correct type after calling aclnnNpuFormatCast. However, when the data type is INT4, you need to use the aclnnConvertWeightToInt4Pack API to convert the data format from ND to NZ and the data type from INT32 to INT4. When FLOAT32 or INT32 is passed, each FLOAT32 or INT32 is identified as eight FLOAT4_E2M1/INT4 internally.
    • scaleOptional supports UINT64, INT64, BFLOAT16, and FLOAT32. offsetOptional and antiquantOffsetOptional are not supported.
    • groupType supports grouping by axis m. Only non-quantization supports ungrouping.
    • quantGroupSize is not supported.
    • actType supports 0, 1, 2, 4, and 5. For details about the constraints, see [object Object]Constraints[object Object].
    • In the non-quantization scenario, the input parameters x and weight and the output parameter out support a maximum of 1024 tensors. In the fake-quantization and full-quantization scenarios, the input parameters x and weight and the output parameter out support a maximum of 128 tensors.

• Return

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]undefined

• Return

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.

[object Object]

• Common constraints

  • If [object Object] is passed: when [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence; when [object Object] is [object Object], [object Object] must be a non-negative sequence and its length cannot be 1; when [object Object] is [object Object], the second column of [object Object] must be a non-negative sequence and its length cannot be 1.
  • The size of each dimension for every tensor in [object Object] and [object Object], after 32-byte alignment, should be less than the maximum value of INT32 (2147483647).
  • actType (int64_t, input for computation): integer parameter, indicating the activation function type. The value ranges from 0 to 5.

• The following input types are supported in non-quantization scenarios:

  • [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT16
  • [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16

• The following input types are supported in quantization scenarios:

  • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: BFLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: BFLOAT16
  • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16
  • [object Object]: INT4; [object Object]: INT4; [object Object]: null; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16 or BFLOAT16 The weight supports the transposed input in the NZ format, that is, the input is in the [E, N, K] format, but the view shape is in the [E, K, N] format to ensure that the operator can identify the transposed status. When the input is transposed, $k/G$ must be 64-byte aligned, K must be 64-byte aligned, and N must be 16-byte aligned. The ND format does not support the transposed input.

• The following input types are supported in fake-quantization scenarios:

  • The shapes of the fake-quantization parameters [object Object] and [object Object] must meet the following requirements ($g$ indicates the number of Matmul groups, $G$ indicates the number of quantization groups, and $G_i$ indicates the number of quantization groups of the i-th tensor).

    [object Object]undefined

  • [object Object]: INT8; [object Object]: INT4; [object Object]: FLOAT32; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: FLOAT32; [object Object]: null This scenario supports symmetric quantization and asymmetric quantization:

    • Symmetric quantization

      • The data type of the output [object Object] is BFLOAT16 or FLOAT16.
      • [object Object] is null.
      • Only the count mode is supported (the operator does not validate [object Object]). [object Object] must be an integer multiple of [object Object] and [object Object] ≤ 18432. [object Object] is the per-group quantization length in the K-axis. Currently, [object Object] is supported.
      • The scale is the result after per-group and per-channel offline fusion. The shape must be $[E, quantGroupNum, N]$, where $quantGroupNum=k \div quantGroupSize$.
      • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[E, N]$.
      • N must be an integer multiple of 8.

    • Asymmetric quantization

      • The data type of the output [object Object] is FLOAT16.
      • Only the count mode is supported (the operator does not validate [object Object]).
      • {k, n} must be {7168, 4096} or {2048, 7168}.
      • The scale is the result after per-group and per-channel offline fusion. The shape must be $[E, 1, N]$.
      • [object Object] is not null. For asymmetric quantization, [object Object] is the auxiliary result of offline computation during the computation process, that is, $antiquantOffset \times scale$. The shape must be $[E, 1, N]$, and the data type must be FLOAT32.
      • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[E, N]$.
      • N must be an integer multiple of 8.

  • In fake-quantization scenarios, if the [object Object] type is INT8, only the per-channel mode is supported. If the [object Object] type is INT4, the per-channel and per-group modes are supported in symmetric quantization. If the per-group mode is used, the number of quantization groups ($G$ or $G_i$) must be exactly divisible by the corresponding $k_i$. For multi-tensor [object Object], the per-group length is defined as $s_i = k_i / G_i$, and all $s_i(i=1,2,...g)$ values must be the same. Asymmetric quantization supports the per-channel mode.

  • In fake-quantization scenarios, if the [object Object] type is INT4, the last dimension of each group of tensors in weight must be an even number. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. In the per-group mode, when [object Object] is transposed, the per-group length $s_i$ must be an even number.

• Supported scenarios for different [object Object] values:

  • In quantization and fake-quantization scenarios, [object Object] can be either [object Object] or [object Object].

  • "S" stands for single-tensor, and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

    [object Object]undefined

[object Object][object Object]

• Common constraints

  • [object Object]: The value can be 0 or 1. When groupListType is set to 0, groupListOptional must be a non-negative monotonic non-decreasing number. When groupListType is set to 1, groupListOptional must be a non-negative number.
  • The size of each dimension for every tensor in [object Object] and [object Object], after 32-byte alignment, should be less than the maximum value of INT32 (2147483647).
  • actType (int64_t, input for computation): integer parameter, indicating the activation function type. The value ranges from 0 to 5.
    • In fake-quantization and non-quantization scenarios, actType can only be set to 0.
    • In full quantization scenarios, when x and weight are of the INT8 type, the quantization mode is static T-C quantization or dynamic K-C quantization, and the scale data type is FLOAT32 or BFLOAT16, actType can be set to 0, 1, 2, 4, or 5. In other full quantization scenarios, actType can only be set to 0.

• Currently, the non-quantization, fake-quantization, and full quantization scenarios are supported.

• The following data types are supported in the non-quantization scenario:

  • The n and k axes of the input weight matrix must be 32-byte aligned.

  • The following input parameters are empty: scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

  • The combinations of data types supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

• The data types supported in the fake-quantization scenario are as follows:

  • The following input parameters are empty: offsetOptional, antiquantOffsetOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

  • The combinations of data types supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

  • Constraints:

    • When x is of type FLOAT8_E4M3FN/FLOAT16/BFLOAT16 and weight is of type FLOAT4_E2M1/FLOAT32, groupSize can only be 32.
    • When x is of type INT8 and weight is of type INT4/INT32, groupSize can only be 128, 192, 256, or 512.
    • The shape of x is fixed to (M, K), and the shape of out is fixed to (M, N).
    • When the types of x and weight are BFLOAT16/FLOAT16 and FLOAT4_E2M1/FLOAT32, or INT8 and INT4/INT32, respectively, only the case where neither x nor weight is transposed is supported. When the types of x and weight are FLOAT8_E4M3FN and FLOAT4_E2M1/FLOAT32, respectively, only the case where x is not transposed and weight is transposed is supported.
    • The transposition of antiquantScale is the same as that of weight.

• The supported input types in static quantization scenarios are as follows:

  • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

  • The following table lists the data type combinations supported by the parameters that are not empty.

    [object Object]undefined

  • The scaleOptional must meet the requirements listed in the following table (g indicates the number of matmul groups, that is, the number of groups).

    [object Object]undefined

• The supported input types in the dynamic quantization (K-T && K-C quantization) scenario are as follows:

  • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional and activationFeatureOutOptional.

  • The combinations of data types supported by non-empty parameters must meet the requirements listed in the following table.

    [object Object]undefined

  • The scaleOptional must meet the requirements listed in the following table (g indicates the number of matmul groups, that is, the number of groups).

    [object Object]undefined

  • perTokenScaleOptional must meet the following requirements:

    [object Object]undefined

• Supported scenarios for different [object Object] values:

  • In the supported scenarios, single indicates a single tensor, and multiple indicates multiple tensors. The sequence is x, weight, out. For example, single-multiple-single indicates that x is a single tensor, weight is multiple tensors, and out is a single tensor.

    [object Object]undefined

[object Object][object Object]

The following example is for reference only. For details, see .

Fake-quantization calling example

Full Quantization Invoking Example