[object Object][object Object][object Object]undefined[object Object]

• Description: Implements grouped matrix multiplication. For example, $y_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g$, where $g$ indicates the number of groups. Currently, M-axis grouping and K-axis grouping are supported. The corresponding functions are as follows:

  • M-axis grouping: $k_i$ and $n_i$ remain consistent for each group, while $m_i$ can vary.
  • K-axis grouping: $m_i$ and $n_i$ remain consistent for each group, while $k_i$ can vary.

• The basic computation formula is as follows (for details, see ):

  $$y_i=x_i\times weight_i + bias_i$$

• Version evolution:

  [object Object]undefined

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor containing the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The superscript "2" in the "Data Type" column of the table above indicates data types that are not supported by the products.
    • FLOAT8_E5M2, FLOAT8_E4M3FN, HIFLOAT8, and FLOAT8_E8M0 are not supported.
    • The input parameter [object Object] does not support BFLOAT16.
    • The input parameter [object Object] does not support INT64.

  • [object Object]Atlas inference products[object Object]: Only the scenario where the data types of [object Object], [object Object], and [object Object] are all FLOAT16 is supported. [object Object] supports only the NZ data format.

• Return

  [object Object] status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]undefined

• Return

  [object Object] status code. For details, see .

[object Object]

[object Object][object Object]

• Based on the precision processing of the input data ([object Object] and [object Object]) and the output matrix ([object Object]) during computation, the GroupedMatmul operator supports three primary scenarios: non-quantization, fake quantization, and full quantization.

  • [object Object]Atlas inference products[object Object]

    [object Object]undefined

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]

    [object Object]undefined

[object Object][object Object]

• Formulas [object Object][object Object]

  • Non-quantization scenario:$$y_i=x_i\times weight_i + bias_i$$

  [object Object][object Object]

  • Full-quantization scenario (without perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32$$y_i=(x_i\times weight_i + bias_i) * scale_i + offset_i$$

  • Full-quantization scenario (with perTokenScaleOptional):

    • [object Object] in INT8 and [object Object] in INT32

      $$y_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i$$

    • [object Object] in INT8 and [object Object] in BFLOAT16

      $$y_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i$$

    • [object Object] in INT4, no [object Object]

      $$y_i=x_i\times (weight_i * scale_i) * per\_token\_scale_i$$

  [object Object][object Object]

  • Fake-quantization scenario:

    • [object Object] in FLOAT16 or BFLOAT16 and [object Object] in INT4 or INT8 (supported only when [object Object], [object Object], and [object Object] are all single-tensor)$$y_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i$$

    [object Object][object Object]

    • [object Object] in INT8 and [object Object] in INT4 (supported only when [object Object], [object Object], and [object Object] are all single-tensor) ([object Object] is a required parameter. It is an auxiliary result of offline computation. It is defined as $bias_i=8\times weight_i * scale_i$, reduced along the K-axis.)$$y_i=((x_i - 8) \times weight_i * scale_i+bias_i ) * per\_token\_scale_i$$

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.

[object Object]

• Common constraints

[object Object][object Object] - If [object Object] and [object Object] need to be transposed, the corresponding tensors must be . - The size of the last dimension for each tensor in [object Object] and [object Object] should be less than 65536. The last dimension of $x_i$ refers to the K-axis when [object Object] is not transposed or the M-axis when [object Object] is transposed. The last dimension of $weight_i$ refers to the N-axis when [object Object] is not transposed or the K-axis when [object Object] is transposed. - When the [object Object] is FRACTAL_NZ, the shape of [object Object] must meet the requirements of the FRACTAL_NZ format. - Generally, [object Object] supports only 1D, and the length must match the M-axis size of [object Object]. This parameter only supports scenarios where [object Object], [object Object], and [object Object] are all single-tensor (with a TensorList length of 1). - When the length of the TensorList in the output is 1, [object Object] constrains the valid portion of the output data. Any portion not specified in [object Object] will not be updated. - When [object Object] is [object Object], [object Object] must be a non-negative, monotonically non-decreasing sequence, representing the cumulative sum (cumsum) results of the grouping axis sizes. When [object Object] is [object Object], it must be a non-negative sequence representing the size of each group along the grouping axis. When [object Object] is [object Object], it must be a non-negative sequence with a shape of [E, 2], where $E$ represents the group size. The data layout is [object Object], where [object Object] indicating the size of each group along the grouping axis. For details, see . - [object Object] indicates the axis to be grouped. For example, if the matrix multiplication is [object Object], [object Object] has the following options: [object Object] means no axis grouping, [object Object] indicates M-axis grouping, [object Object] indicates N-axis grouping, and [object Object] indicates K-axis grouping. Currently, N-axis grouping is not supported. For details, see [object Object]groupType constraints[object Object]. - [object Object] (int64_t, computation input): integer type, indicating the activation function type. The value ranges from 0 to 5. The supported enumerated values are as follows: * 0: GMMActType::GMM_ACT_TYPE_NONE * 1: GMMActType::GMM_ACT_TYPE_RELU * 2: GMMActType::GMM_ACT_TYPE_GELU_TANH * 3: GMMActType::GMM_ACT_TYPE_GELU_ERR_FUNC (not supported) * 4: GMMActType::GMM_ACT_TYPE_FAST_GELU * 5: GMMActType::GMM_ACT_TYPE_SILU

[object Object]

[object Object][object Object]

[object Object]

• Product specifications

  • [object Object]: integer type, indicating the axis to be grouped. Currently, only M-axis grouping is supported.
  • [object Object]: The value can be 0 or 1. [object Object] indicates that the values in [object Object] are non-negative, monotonically non-decreasing numbers, representing the cumulative sum (cumsum) results of the grouping axis sizes. [object Object] indicates that the values in [object Object] are non-negative numbers, representing the size of each group along the grouping axis.
  • [object Object]: Currently, only [object Object] is supported, indicating [object Object].
  • [object Object]: This parameter is not supported.
  • The input and output support only the FLOAT16 type. The N-axis size of the output [object Object] must be a multiple of 16.

• Supported scenarios

  [object Object]undefined

[object Object][object Object]

The following example is for reference only. For details, see .