[object Object][object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication, supporting non-uniform matrix dimension sizes across multiple groups. The basic function is matrix multiplication, for example, $y_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups, and $m_i$, $k_i$, and $n_i$ indicate the corresponding dimension sizes. Both input and output parameters are of the aclTensorList type, with the following functions:

  • K-axis grouping: $k_i$ varies across groups, while $m_i$ and $n_i$ remain the same for each group. In this case, $x_i$ and $weight_i$ can be concatenated along the K-axis.
  • M-axis grouping: $k_i$ remains the same for each group. In this case, $weight_i$ and $y_i$ can be concatenated along the N-axis.

  Compared with , this API has the following new features:

  • The values in [object Object] can be the sizes of groups along the grouping axis.
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
    • Static quantization (per-tensor and per-channel), BFLOAT16 and FLOAT16 outputs, with or without activation (For details, refer to . Same below.)
    • Dynamic quantization (per-tensor and per-channel), BFLOAT16 and FLOAT16 outputs, with or without activation
    • Fake-quantization with INT4 input [object Object] without activation in per-channel and per-group modes

  Notes:

  • "Single-tensor" means that tensors of all groups in a tensor list are concatenated into one tensor along the axis specified [object Object].
  • Tensor transpose: If the tensor shape is [M, K], the stride is [1, M], and the data layout is [K, M], then the tensor is a non-contiguous tensor.

• Formula:

  • Non-quantization scenario:
  $$y_i=x_i\times weight_i + bias_i$$

  • Quantization scenario (static quantization, T-C && T-T, without perTokenScaleOptional):

    $$y_i=(x_i\times weight_i) * scale_i + offset_i$$
    • [object Object] in INT8 and [object Object] in INT32
    $$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$
    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32, without offset
    $$y_i=(x_i\times weight_i) * scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, T-T && T-C && K-T && K-C):

    $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i$$
    • [object Object] in INT8 and [object Object] in INT32
    $$y_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i$$
    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32
    $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

  • Quantization scenario (dynamic quantization, MX && G-B):

    $$y_i[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (xSlice_i * weightSlice_i)) * (per\_token\_scale_i[m/gsM, j] * scale_i[j, n/gsN])) + bias_i[n]$$

    $gsM$, $gsN$, and $gsK$ represent the quantization block sizes for the M, N, and K axes respectively. $xSlice_i$ denotes a vector of length $gsK$ from the $m$-th row of $x_i$, and $weightSlice_i$ denotes a vector of length $gsK$ from the $n$-th column of $weight_i$. Both tensors are sliced along the K-axis starting from $j \times gsK$, where $j \in [0, kLoops)$ and $kLoops = \lceil K_i / gsK \rceil$. Additionally, a final slice length less than $gsK$ is supported.

  • Fake-quantization scenario:

  $$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  • x (aclTensorList *, computation input): aclTensorList on the device, $x$ in the formula. The can be ND, and the maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, FLOAT32, INT8, or INT4.
    • [object Object]Atlas inference products[object Object]: The data type can be FLOAT16.
  • weight (aclTensorList *, computation input): aclTensorList on the device, $weight$ in the formula. The maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, FLOAT32, INT8, or INT4, and the can be ND or FRACTAL_NZ.
    • [object Object]Atlas inference products[object Object]: The data type can be FLOAT16, and the can only be FRACTAL_NZ.
  • biasOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, $bias$ in the formula. The can be ND, and the length is the same as that of [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16, FLOAT32, or INT32.
    • [object Object]Atlas inference products[object Object]: The data type can be FLOAT16.
  • scaleOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the scale factor for quantization parameters. The can be ND. Generally, the length is the same as that of [object Object]. For details about the constraints, see .
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be UINT64, BFLOAT16, or FLOAT32.
    • [object Object]Atlas inference products[object Object]: This parameter is not supported currently and needs to be passed as a null pointer.
  • offsetOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the offset for quantization parameters. The can be ND, and the length is the same as that of [object Object].
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT32.
    • [object Object]Atlas inference products[object Object]: This parameter is not supported currently and needs to be passed as a null pointer.
  • antiquantScaleOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the scale factor for fake-quantization parameters. The can be ND, and the length is the same as that of [object Object]. For details about the constraints, see .
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference products[object Object]: This parameter is not supported currently and needs to be passed as a null pointer.
  • antiquantOffsetOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the offset for fake-quantization parameters. The can be ND, and the length is the same as that of [object Object]. For details about the constraints, see .
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference products[object Object]: This parameter is not supported currently and needs to be passed as a null pointer.
  • perTokenScaleOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the scale factor (introduced by [object Object] quantization) for quantization parameters. The can be ND. This parameter only supports scenarios where [object Object], [object Object], and [object Object] are all single-tensor (with a TensorList length of 1). For details about the constraints, see .
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT32.
    • [object Object]Atlas inference products[object Object]: This parameter is not supported and needs to be passed as a null pointer.
  • groupListOptional (aclTensor *, computation input): optional parameter, aclTensor type on the device, indicating the Matmul size distribution along the grouping axis for inputs and outputs. The data type can be INT64, and the can be ND. Note that when the length of the TensorList in the output is 1, the last value in [object Object] constrains the valid portion of the output data. Any portion not specified in [object Object] will not be updated.
  • activationInputOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device, indicating the backward input of the activation function. Currently, only [object Object] is supported.
  • activationQuantScaleOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device. Currently, only [object Object] is supported.
  • activationQuantOffsetOptional (aclTensorList *, computation input): optional parameter, aclTensorList on the device. Currently, only [object Object] is supported.
  • splitItem (int64_t, computation input): integer type, indicating whether tensor splitting is required for the output. [object Object] or [object Object] indicates multi-tensor, and [object Object] or [object Object] indicates single-tensor.
  • groupType (int64_t, computation input): integer type, indicating the axis to be grouped. For example, if the matrix multiplication is [object Object], [object Object] has the following options: [object Object] means no axis grouping, [object Object] indicates M-axis grouping, [object Object] indicates N-axis grouping, and [object Object] indicates K-axis grouping.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Currently, N-axis grouping is not supported.
    • [object Object]Atlas inference products[object Object]: Currently, only M-axis grouping is supported.
  • groupListType (int64_t, computation input): integer type. The value can be: *0: Values in [object Object] are the cumulative sum (cumsum) results of the grouping axis sizes. * 1: Values in [object Object] are the sizes of groups along the grouping axis. *2: The shape of [object Object] is [e, 2], where [object Object] indicates the group size. The data layout is [object Object], where [object Object] is the size of each group along the grouping axis. * [object Object]Atlas inference products[object Object]: The value [object Object] is not supported. * [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value [object Object] is supported only when the input type of [object Object] and [object Object] is INT8 and [object Object] is set to [object Object] (M-axis grouping).
  • actType (int64_t, computation input): integer type, indicating the activation function type.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value ranges from 0 to 5, corresponding to the following enumerated values:
      • 0: GMMActType::GMM_ACT_TYPE_NONE
      • 1: GMMActType::GMM_ACT_TYPE_RELU
      • 2: GMMActType::GMM_ACT_TYPE_GELU_TANH
      • 3: GMMActType::GMM_ACT_TYPE_GELU_ERR_FUNC (not supported)
      • 4: GMMActType::GMM_ACT_TYPE_FAST_GELU
      • 5: GMMActType::GMM_ACT_TYPE_SILU
    • [object Object]Atlas inference products[object Object]: Currently, only [object Object] is supported, indicating [object Object].
  • out (aclTensorList *, computation output): aclTensorList on the device, $y$ in the formula. The can be ND, and the maximum length is 128.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The data type can be FLOAT16, BFLOAT16, INT8, FLOAT32, or INT32.
    • [object Object]Atlas inference products[object Object]: The data type can be FLOAT16.
  • activationFeatureOutOptional (aclTensorList *, computation output): aclTensorList on the device, the input data of the activation function. Currently, only [object Object] is supported.
  • dynQuantScaleOutOptional (aclTensorList *, computation output): aclTensorList on the device. Currently, only [object Object] is supported.
  • workspaceSize (uint64_t *, output): size of the workspace to be allocated on the device.
  • executor (aclOpExecutor **, output): operator executor, containing the operator computation process.

• Return

  [object Object] status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• If [object Object] is passed: when [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence; when [object Object] is [object Object], [object Object] must be a non-negative sequence; when [object Object] is [object Object], the second column of [object Object] must be a non-negative sequence and its length cannot be 1.

• The size of each dimension for every tensor in [object Object] and [object Object], after 32-byte alignment, should be less than the maximum value of INT32 (2147483647).

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • The following input types are supported in non-quantization scenarios:

    • [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT16
    • [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16
    • [object Object]: FLOAT32; [object Object]: FLOAT32; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT32 (supported only when [object Object], [object Object], and [object Object] are all single-tensor)

  • The following input types are supported in quantization scenarios:

    • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: INT8
    • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: BFLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: BFLOAT16
    • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16
    • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: INT32
    • [object Object]: INT4; [object Object]: INT4; [object Object]: null; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16
    • [object Object]: INT4; [object Object]: INT4; [object Object]: null; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: BFLOAT16

  • The following input types are supported in fake-quantization scenarios:

    • [object Object]: FLOAT16; [object Object]: INT8 or INT4; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: FLOAT16

    • The shapes of the fake-quantization parameters [object Object] and [object Object] must meet the following requirements ($g$ indicates the number of Matmul groups, $G$ indicates the number of quantization groups, and $G_i$ indicates the number of quantization groups of the i-th tensor).

      [object Object]undefined

    • [object Object]: BFLOAT16; [object Object]: INT8 or INT4; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16

    • [object Object]: INT8; [object Object]: INT4; [object Object]: FLOAT32; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: FLOAT32; [object Object]: null This scenario supports symmetric quantization and asymmetric quantization:

      • Symmetric quantization
        • In this case, the data type of the output [object Object] is BFLOAT16 or FLOAT16.
        • In this case, [object Object] is null.
        • In this case, only the count mode is supported (the operator does not validate [object Object]). [object Object] must be an integer multiple of [object Object] and [object Object] ≤ 18432. [object Object] is the per-group quantization length in the K-axis. Currently, [object Object] is supported.
        • In this case, the scale is the result after per-group and per-channel offline fusion. The shape must be $[e, quantGroupNum, n]$, where $quantGroupNum=k \div quantGroupSize$.
        • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[e, n]$.
        • In this case, [object Object] must be an integer multiple of 8.
      • Asymmetric quantization
        • In this case, the data type of the output [object Object] is FLOAT16.
        • In this case, only the count mode is supported (the operator does not validate [object Object]).
        • In this case, {k, n} must be {7168, 4096} or {2048, 7168}.
        • The scale is the result after per-group and per-channel offline fusion. The shape must be $[e, 1, n]$.
        • In this case, [object Object] is not null. For asymmetric quantization, [object Object] is the auxiliary result of offline computation during the computation process, that is, $antiquantOffset \times scale$. The shape must be $[e, 1, n]$, and the data type must be FLOAT32.
        • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[e, n]$.
        • In this case, [object Object] must be an integer multiple of 8.

  • In quantization scenarios, if the [object Object] type is INT4, the following constraints must be met ($g$ indicates the number of Matmul groups and $G$ indicates the number of groups partitioned along the K-axis for per-group quantization):

    • If the data format of [object Object] is ND, [object Object] must be an integer multiple of 8.
    • Per-channel and per-group quantization are supported. In the per-channel scenario, the shape of the scale must be $[g, n]$. In the per-group scenario, the shape must be $[g, G, n]$.
    • In the per-group scenario, $G$ must be exactly divisible by $k$, and $k/G$ must be an even number.
    • In this scenario, only [object Object] ([object Object], [object Object], and [object Object] are all single-tensor), [object Object], and [object Object] are supported.
    • Weight transposition is not supported in this scenario.

  • In fake-quantization scenarios, if the [object Object] type is INT8, only the per-channel mode is supported. If the [object Object] type is INT4, the per-channel and per-group modes are supported in symmetric quantization. If the per-group mode is used, the number of quantization groups ($G$ or $G_i$) must be exactly divisible by the corresponding $k_i$. For multi-tensor [object Object], the per-group length is defined as $s_i = k_i / G_i$, and all $s_i(i=1,2,...g)$ values must be the same. Asymmetric quantization supports the per-channel mode.

  • In fake-quantization scenarios, if the [object Object] type is INT4, the last dimension of each group of tensors in weight must be an even number. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. In the per-group mode, when [object Object] is transposed, the per-group length $s_i$ must be an even number.

  • Supported scenarios for different [object Object] values:

    • In quantization and fake-quantization scenarios, [object Object] can be either [object Object] or [object Object].

    • "S" stands for single-tensor, and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

      [object Object]undefined

  • The size of the last dimension for each tensor in [object Object] and [object Object] should be less than 65536. The last dimension of $x_i$ refers to the K-axis when [object Object] is not transposed or the M-axis when [object Object] is transposed. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed.

  • Activation function computation is supported only in quantization (per-token) and dequantization scenarios.

• [object Object]Atlas inference products[object Object]:

  • The input and output support only the FLOAT16 type. The N-axis size of the output [object Object] must be a multiple of 16.

    [object Object]undefined

[object Object]

The following example is for reference only. For details, see .