[object Object][object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication, supporting non-uniform matrix dimension sizes across multiple groups. The basic function is matrix multiplication, for example, $y_i[m_i,n_i]=x_i[m_i,k_i] times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups and $m_i$, $k_i$, and $n_i$ define the shapes for each group. Both inputs and outputs are of the aclTensorList type, with the following functions:

  • K-axis grouping: $k_i$ varies across groups, while $m_i$ and $n_i$ remain the same for each group. In this case, $x_i$ and $weight_i$ can be concatenated along the K-axis.
  • M-axis grouping: $k_i$ remains the same for each group. In this case, $weight_i$ and $y_i$ can be concatenated along the N-axis.

  New features compared with :

  • The input [object Object] data supports the AI processor-affinity layout format (FRACTAL_NZ).
  • The [object Object] parameter is added. It is an integer that indicates the quantization group size in per-group mode. If per-group quantization is not involved, set this parameter to [object Object].

• Formulas:

  [object Object][object Object]

  • Non-quantization scenario:$$y_i=x_i \times weight_i + bias_i$$

  [object Object][object Object]

  • Quantization scenario (without perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i \times weight_i + bias_i) * scale_i + offset_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32, without offset

      $$y_i=(x_i \times weight_i) * scale_i + bias_i$$

  • Quantization scenario (with perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i \times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16/FLOAT16/FLOAT32

      $$y_i=(x_i \times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

  • Quantization scenario (MX quantization, without bias or activation layer):

    $$y_i=(x_i \times per\_token\_scale_i) * (weight_i \times scale_i)$$

  [object Object][object Object]

  • Dequantization scenario:$$y_i=(x_i \times weight_i + bias_i) * scale_i$$

  [object Object][object Object]

  • Fake-quantization scenario:$$y_i=x_i \times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
    • The superscript "1" in the "Data Type" column of the table above indicates data types that are supported by the products, and the superscript "2" indicates data types that are not supported by the products.
    • The [object Object] format can be converted from ND to NZ by calling [object Object] and [object Object]. When the input data type is INT32, each INT32 value is treated as eight INT4 values inside the API.
  • [object Object]Atlas inference products[object Object]:
    • Only FLOAT16 is supported. [object Object] supports only the FRACTAL_NZ format, which needs to be converted through auxiliary APIs.
    • Quantization/asymmetric quantization parameters such as [object Object] and [object Object] are not supported, and need to be passed as null pointers.
    • [object Object] supports only M-axis grouping ([object Object]). The value of [object Object] can only be [object Object]. [object Object] is not supported.

• Return

  [object Object] status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]undefined

• Return

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.
• Common constraints
  • If [object Object] is passed: when [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence; when [object Object] is [object Object], [object Object] must be a non-negative sequence and its length cannot be 1; when [object Object] is [object Object], the second column of [object Object] must be a non-negative sequence and its length cannot be 1.
  • The size of each dimension for every tensor in [object Object] and [object Object], after 32-byte alignment, should be less than the maximum value of INT32 (2147483647).

[object Object]

• The following input types are supported in non-quantization scenarios:

  • [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: FLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: FLOAT16
  • [object Object]: BFLOAT16; [object Object]: BFLOAT16; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: BFLOAT16

• The following input types are supported in quantization scenarios:

  • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: BFLOAT16; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: BFLOAT16
  • [object Object]: INT8; [object Object]: INT8; [object Object]: INT32; [object Object]: FLOAT32; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16
  • [object Object]: INT4; [object Object]: INT4; [object Object]: null; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: null; [object Object]: null or FLOAT32; [object Object]: null; [object Object]: FLOAT16 or BFLOAT16

• The following input types are supported in fake-quantization scenarios:

  • The shapes of the fake-quantization parameters [object Object] and [object Object] must meet the following requirements ($g$ indicates the number of Matmul groups, $G$ indicates the number of quantization groups, and $G_i$ indicates the number of quantization groups of the i-th tensor).

    [object Object]undefined

  • [object Object]: INT8; [object Object]: INT4; [object Object]: FLOAT32; [object Object]: UINT64; [object Object]: null; [object Object]: null; [object Object]: FLOAT32; [object Object]: null This scenario supports symmetric quantization and asymmetric quantization:

    • Symmetric quantization

      • The data type of the output [object Object] is BFLOAT16 or FLOAT16.
      • [object Object] is null.
      • Only the count mode is supported (the operator does not validate [object Object]). [object Object] must be an integer multiple of [object Object] and [object Object] ≤ 18432. [object Object] is the per-group quantization length in the K-axis. Currently, [object Object] is supported.
      • The scale is the result after per-group and per-channel offline fusion. The shape must be $[E, quantGroupNum, N]$, where $quantGroupNum=k \div quantGroupSize$.
      • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[E, N]$.
      • N must be an integer multiple of 8.

    • Asymmetric quantization

      • The data type of the output [object Object] is FLOAT16.
      • Only the count mode is supported (the operator does not validate [object Object]).
      • {k, n} must be {7168, 4096} or {2048, 7168}.
      • The scale is the result after per-group and per-channel offline fusion. The shape must be $[E, 1, N]$.
      • [object Object] is not null. For asymmetric quantization, [object Object] is the auxiliary result of offline computation during the computation process, that is, $antiquantOffset \times scale$. The shape must be $[E, 1, N]$, and the data type must be FLOAT32.
      • The bias is the auxiliary result of offline computation during the computation process. Its value must be $8\times weight \times scale$ and is accumulated in the first dimension. The shape must be $[E, N]$.
      • N must be an integer multiple of 8.

  • In fake-quantization scenarios, if the [object Object] type is INT8, only the per-channel mode is supported. If the [object Object] type is INT4, the per-channel and per-group modes are supported in symmetric quantization. If the per-group mode is used, the number of quantization groups ($G$ or $G_i$) must be exactly divisible by the corresponding $k_i$. For multi-tensor [object Object], the per-group length is defined as $s_i = k_i / G_i$, and all $s_i(i=1,2,...g)$ values must be the same. Asymmetric quantization supports the per-channel mode.

  • In fake-quantization scenarios, if the [object Object] type is INT4, the last dimension of each group of tensors in weight must be an even number. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. In the per-group mode, when [object Object] is transposed, the per-group length $s_i$ must be an even number.

• Supported scenarios for different [object Object] values:

  • In quantization and fake-quantization scenarios, [object Object] can be either [object Object] or [object Object].

  • "S" stands for single-tensor, and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

    [object Object]undefined

[object Object][object Object]

• The input and output support only the FLOAT16 type. The N-axis size of the output [object Object] must be a multiple of 16.

  "S" stands for single-tensor and "M" stands for multi-tensor, expressed in the sequence of [object Object], [object Object], [object Object]. For example, "SMS" indicates single-tensor [object Object], multi-tensor [object Object], and single-tensor [object Object].

  [object Object]undefined

[object Object][object Object]

The following example is for reference only. For details, see .