Documentation
Download

aclnnGroupedMatmulV4

Notice: This API will be deprecated in later versions. Use the latest aclnnGroupedMatmulV5 API instead.

Applicable Products

ProductSupported
Ascend 950PR/Ascend 950DT√
Atlas A3 training products/Atlas A3 inference products√
Atlas A2 training products/Atlas A2 inference products√
Atlas 200I/500 A2 inference products×
Atlas inference products√
Atlas training products×

Function

  • Description: Implements grouped matrix multiplication, where the dimension sizes of each group can differ. The basic function is matrix multiplication, as in yi[mi,ni]=xi[mi,ki]×weighti[ki,ni],i=1...gy_i[m_i,n_i]=x_i[m_i,k_i] \times weight_i[k_i,n_i], i=1...g, where g is the number of groups and mi/ki/nim_i/k_i/n_i are the corresponding dimensions. Both input and output parameter types are aclTensorList, with the following functions:

    • k-axis grouping: kik_i values differ, but mi/nim_i/n_i are the same for each group. In this case, xi/weightix_i/weight_i can be concatenated along kik_i.

    • m-axis grouping: kik_i is the same for each group, and weighti/yiweight_i/y_i can be concatenated on nin_i.

    Compared with the GroupedMatmulV3 API, this API adds the following:

    • Supports using the value in groupListOptional as the size of each group on the grouping axis.

    • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

      • Supports static quantization (pertensor+perchannel) (for the quantization mode, see Quantization Introduction, the same below) with BFLOAT16 and FLOAT16 output, in scenarios with and without activation.

      • Supports dynamic quantization (pertoken+perchannel) with BFLOAT16 and FLOAT16 output, in scenarios with and without activation.

      • Supports pseudo-quantization with INT4 weight input, without activation, supporting both per-channel and per-group modes.

    • Ascend 950PR/Ascend 950DT:

      • Supports static quantization (1. per-tensor–per-channel (T-C); 2. per-tensor–per-tensor (T-T)) with BFLOAT16, FLOAT16, and FLOAT32 output, with bias.

      • Supports dynamic quantization (1. per-token–per-channel (K-C); 2. per-token–per-tensor (K-T); 3. per-tensor–per-tensor (T-T); 4. per-tensor–per-channel (T-C); 5. MX quantization; 6. per-group–per-block (G-B)) with BFLOAT16, FLOAT16, and FLOAT32 output, with bias.

      • Supports pseudo-quantization with INT4, FLOAT8_E5M2, FLOAT8_E4M3FN, and HIFLOAT8 weight input, without activation, supporting only per-channel mode.

NOTE

  • Single tensor means that all grouped tensors in a tensor list are merged into one on the grouping axis specified by groupType; otherwise, it is multi-tensor.

  • Tensor transposition: refers to the scenario where the tensor shape is [M, K], the stride is [1, M], and the data layout is [K, M], i.e., a non-contiguous tensor.

  • Formula:

    • Non-quantization scenario:

      yi=xi×weighti+biasiy_i=x_i\times weight_i + bias_i

    • Quantization scenario (static quantization, T-C && T-T quantization, without perTokenScaleOptional):

      yi=(xi×weighti)∗scalei+offsetiy_i=(x_i\times weight_i) * scale_i + offset_i

      • x is INT8, bias is INT32

        yi=(xi×weighti+biasi)∗scalei+offsetiy_i=(x_i\times weight_i + bias_i) * scale_i + offset_i

      • x is INT8, bias is BFLOAT16/FLOAT16/FLOAT32, without offset

        yi=(xi×weighti)∗scalei+biasiy_i=(x_i\times weight_i) * scale_i + bias_i

    • Quantization scenario (dynamic quantization, T-T && T-C && K-T && K-C quantization):

      yi=(xi×weighti)∗scalei∗per_token_scaleiy_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i

      • x is INT8, bias is INT32

        yi=(xi×weighti+biasi)∗scalei∗per_token_scaleiy_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i

      • x is INT8, and bias is BFLOAT16/FLOAT16/FLOAT32.

        yi=(xi×weighti)∗scalei∗per_token_scalei+biasiy_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i

    • Quantization scenario (dynamic quantization, MX && G-B quantization):

      yi[m,n]=∑j=0kLoops−1((∑k=0gsK−1(xSlicei∗weightSlicei))∗(per_token_scalei[m/gsM,j]∗scalei[j,n/gsN]))+biasi[n]y_i[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (xSlice_i * weightSlice_i)) * (per\_token\_scale_i[m/gsM, j] * scale_i[j, n/gsN])) + bias_i[n]

      Where gsM, gsN, and gsK represent the quantization block sizes of the M/N/K axes respectively, xSliceixSlice_i represents the vector of length gsK in the m-th row of xix_i, weightSliceiweightSlice_i represents the vector of length gsK in the n-th column of weightiweight_i, the K axis is sliced starting from j * gsK, j has a value range of [0, kLoops), kLoops=ceil(KiK_i / gsK), and the last slice is allowed to be shorter than gsK.

    • Pseudo-quantization scenario:

      yi=xi×(weighti+antiquant_offseti)∗antiquant_scalei+biasiy_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnGroupedMatmulV4GetWorkspaceSize" API to obtain the input parameters and calculate the required workspace size based on the computation process, and then call the "aclnnGroupedMatmulV4" API to perform the computation.

aclnnStatus aclnnGroupedMatmulV4GetWorkspaceSize(
  const aclTensorList *x,
  const aclTensorList *weight,
  const aclTensorList *biasOptional,
  const aclTensorList *scaleOptional,
  const aclTensorList *offsetOptional,
  const aclTensorList *antiquantScaleOptional,
  const aclTensorList *antiquantOffsetOptional,
  const aclTensorList *perTokenScaleOptional,
  const aclTensor     *groupListOptional,
  const aclTensorList *activationInputOptional,
  const aclTensorList *activationQuantScaleOptional,
  const aclTensorList *activationQuantOffsetOptional,
  int64_t              splitItem,
  int64_t              groupType,
  int64_t              groupListType,
  int64_t              actType,
  aclTensorList       *out,
  aclTensorList       *activationFeatureOutOptional,
  aclTensorList       *dynQuantScaleOutOptional,
  uint64_t            *workspaceSize,
  aclOpExecutor       **executor)
aclnnStatus aclnnGroupedMatmulV4(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnGroupedMatmulV4GetWorkspaceSize

  • Parameters:

    Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-Contiguous Tensor
    x (aclTensorList) Input Represents input x in the formula. tensorList length support: [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, FLOAT32, INT8, INT4, FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 ND - -
    weight (aclTensorList) Input weight in the formula. tensorList length support: [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, FLOAT32, INT8, INT4, FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, FLOAT4_E2M1 ND, FRACTAL_NZ - -
    biasOptional (aclTensorList) Input Represents the bias in the formula. Has the same length as weight. INT32, BFLOAT16, FLOAT16, FLOAT32, INT8 ND - -
    scaleOptional (aclTensorList) Input Represents the scale factor in the quantization parameter. In general, the length is the same as weight. UINT64, INT64, BFLOAT16, FLOAT32, FLOAT8_E8M0 ND - -
    offsetOptional (aclTensorList) Input Represents the offset in the quantization parameter. The length is the same as that of weight. FLOAT32 ND - -
    antiquantScaleOptional（aclTensorList） Input Represents the scale factor in the pseudo-quantization parameter. The length is the same as that of weight. FLOAT16, BFLOAT16 ND - -
    antiquantOffsetOptional (aclTensorList) Input Represents the offset in the pseudo-quantization parameter. Same length as weight. FLOAT16, BFLOAT16 ND - -
    perTokenScaleOptional (aclTensorList) Input Represents the scale factor introduced by x quantization in the quantization parameter. Only supports the case where x, weight, and out are all single tensors. FLOAT32, FLOAT8_E8M0 ND - -
    groupListOptional (aclTensorList) Input Represents the matmul size distribution along the group axis direction of the input and output. - INT64 ND - -
    activationInputOptional (aclTensorList) Input Currently, only nullptr is supported. Currently, only nullptr is supported. - - - -
    activationQuantScaleOptional (aclTensorList) Input Currently only nullptr is supported. Currently only nullptr is supported. - - - -
    activationQuantOffsetOptional (aclTensorList) Input Currently only nullptr is supported. Currently, only nullptr is supported. - - - -
    splitItem (int64_t) Input. Integer parameter that represents whether the output is split into multiple tensors. 0/1 represents output as multiple tensors; 2/3 represents output as a single tensor. - - - -
    groupType (int64_t) Input Integer parameter that represents the axis to be grouped. Enumeration values: -1, 0, 2. If the matrix multiplication is C[m,n]=A[m,k]xB[k,n], then groupType value -1: no grouping, 0: grouping along the m axis, 2: grouping along the k axis. - - - -
    groupListType (int64_t) Input - Enumeration values: 0, 1, 2. For comprehensive constraints, see Constraints. - - - -
    actType (int64_t) Input Represents the activation function type. Value range: 0 to 5.
    0: GMM_ACT_TYPE_NONE;
    1: GMM_ACT_TYPE_RELU;
    2: GMM_ACT_TYPE_GELU_TANH;
    3: GMM_ACT_TYPE_GELU_ERR_FUNC;
    4: GMM_ACT_TYPE_FAST_GELU;
    5: GMM_ACT_TYPE_SILU;
    For comprehensive constraints, see Constraints.    		 INT64 - - -
    out (aclTensorList) Output The output y in the formula. Supports tensorList length in the range [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, INT8, FLOAT32, INT32 ND - -
    activationFeatureOutOptional (aclTensorList) Output Input data of the activation function. Currently only supports passing nullptr. - - - -
    dynQuantScaleOutOptional (aclTensorList) Output Currently only nullptr is supported. Currently only nullptr is supported. - - - -
    workspaceSize (uint64_t) Output Returns the workspace size to be allocated on the Device side. - - - - -
    executor (aclOpExecutor) Output Returns the op executor, which contains the operator computation flow. - - - - -

    • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

      • x supports FLOAT16, BFLOAT16, FLOAT32, INT8, and INT4.

      • weight supports FLOAT16, BFLOAT16, FLOAT32, INT8, and INT4, and supports the ND and FRACTAL_NZ formats.

      • biasOptional supports FLOAT16, FLOAT32, and INT32.

      • scaleOptional supports UINT64, BFLOAT16, and FLOAT32.

      • perTokenScaleOptional supports FLOAT32.

      • out supports FLOAT16, BFLOAT16, INT8, FLOAT32, and INT32.

      • The input parameters x and weight, and the output parameter out support up to 128 tensors.

    • Ascend 950PR/Ascend 950DT:

      • x supports FLOAT8_E4M3FN, FLOAT8_E5M2, INT8, HIFLOAT8, FLOAT16, BFLOAT16, FLOAT32, and FLOAT4_E2M1.

      • weight supports FLOAT8_E4M3FN, FLOAT8_E5M2, INT8, INT4, HIFLOAT8, FLOAT16, BFLOAT16, FLOAT32, and FLOAT4_E2M1, and only the ND format is supported.

      • biasOptional supports INT32, BFLOAT16, FLOAT16, and FLOAT32. When input x is INT8, FLOAT16, BFLOAT16, or FLOAT32, INT32, BFLOAT16, FLOAT16, and FLOAT32 are supported. When input x is FLOAT4_E2M1, only FLOAT32 is supported. For other input types, a null pointer must be passed.

      • scaleOptional supports UINT64, INT64, BFLOAT16, FLOAT32, and FLOAT8_E8M0.

      • perTokenScaleOptional supports FLOAT32 and FLOAT8_E8M0.

      • groupListType does not support the value 2.

      • actType supports 0, 1, 2, 4, and 5. For comprehensive constraints, see Constraints.

      • out supports BFLOAT16, FLOAT16, and FLOAT32.

      • offsetOptional is not supported.

      • groupType supports m-axis grouping. Only non-quantization and quantization support k-axis grouping. Only non-quantization and pseudo-quantization support no grouping.

      • In the non-quantization scenario, the input parameters x and weight and the output parameter out support up to 1024 tensors; in the pseudo-quantization scenario, they support up to 128 tensors; in the full quantization scenario, only a single tensor is supported.

  • Return value

    aclnnStatus: return code. For details, see aclnn Return Code.

    The first-phase API performs input parameter validation and returns an error in the following scenarios:

    Return Value Error Code Description
    ACLNN_ERR_PARAM_NULLPTR 161001 The input parameter is a required input, output, or required attribute and is a null pointer.
    ACLNN_ERR_PARAM_INVALID 161002 The data types and data formats of x, weight, biasOptional, scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, groupListOptional, and out are not within the supported range.
    The length of weight is not within the supported range.
    If bias is not empty, the length of bias is not equal to the length of weight.
    The dimension of groupListOptional is 1.
    In scenarios where splitItem is 2 or 3, the length of out is not equal to 1.
    In scenarios where splitItem is 0 or 1, the length of out is not equal to the length of weight, and the length of groupListOptional is not equal to the length of weight.

aclnnGroupedMatmulV4

  • Parameters:

    Parameters Input/Output Description
    workspace Input Workspace memory address applied for on the Device side.
    workspaceSize Input Workspace size applied for on the Device side, obtained through the first-phase API aclnnGroupedMatmulV4GetWorkspaceSize.
    executor Input Operator executor, which contains the operator computation flow.
    stream Input Specifies the stream for executing the task.

  • Return value

    aclnnStatus: return code. For details, see aclnn Return Code.

Scenario Classification

  • Formula

    • Non-quantization scenario:yi=xi×weighti+biasiy_i=x_i\times weight_i + bias_i

    • Full quantization scenario (without perTokenScaleOptional):

      • x is INT8, bias is INT32yi=(xi×weighti+biasi)∗scalei+offsetiy_i=(x_i\times weight_i + bias_i) * scale_i + offset_i

    • Full quantization scenario (with perTokenScaleOptional):

      • x is INT8, bias is INT32

        yi=(xi×weighti+biasi)∗scalei∗per_token_scaleiy_i=(x_i\times weight_i + bias_i) * scale_i * per\_token\_scale_i

      • x is INT8, bias is BFLOAT16

        yi=(xi×weighti)∗scalei∗per_token_scalei+biasiy_i=(x_i\times weight_i) * scale_i * per\_token\_scale_i + bias_i

      • x is INT4, without bias

        yi=xi×(weighti∗scalei)∗per_token_scaleiy_i=x_i\times (weight_i * scale_i) * per\_token\_scale_i

    • Pseudo-quantization scenario:

      • x is Float16 or BFloat16, and weight is INT4 or INT8 (only the scenario where x, weight, and y are all single tensors is supported).yi=xi×(weighti+antiquant_offseti)∗antiquant_scalei+biasiy_i=x_i\times (weight_i + antiquant\_offset_i) * antiquant\_scale_i + bias_i

      • x is INT8, and weight is INT4 (only the single-tensor scenario for x, weight, and y is supported). bias is a mandatory parameter, which is an offline-computed auxiliary result, and biasi=8×weighti∗scaleibias_i=8\times weight_i * scale_i, reduced along the K axis.yi=((xi−8)×weighti∗scalei+biasi)∗per_token_scaleiy_i=((x_i - 8) \times weight_i * scale_i+bias_i ) * per\_token\_scale_i

Constraints

  • Deterministic computation:

    • aclnnGroupedMatmulV4 defaults to a deterministic implementation.
Atlas A2 training products/Atlas A2 inference products、 Atlas A3 training products/Atlas A3 inference products
  • General Constraints

- If x and weight need to be transposed, the transposed tensor must be non-contiguous.

- The last dimension size of each group of tensors in x and weight must be less than 65536. The last dimension of $x_i$ refers to the K axis of $x_i$ when x is not transposed, or the M axis of $x_i$ when x is transposed. The last dimension of $weight_i$ refers to the N axis of $weight_i$ when weight is not transposed, or the K axis of $weight_i$ when weight is transposed.

- When the weight [data format](../common/data_format.md) is FRACTAL_NZ, the shape of weight must meet the FRACTAL_NZ format requirements.

- perTokenScaleOptional: Generally, only 1D with a length equal to the M of x is supported. Only the scenario where x, weight, and out are all single tensors (TensorList length is 1) is supported.

- groupListOptional: When the length of the TensorList in the output is 1, groupListOptional constrains the valid portion of the output data. The parts not specified in groupListOptional will not be updated.

- When groupListType is 0, the values in groupListOptional must be a non-negative monotonic non-decreasing sequence, representing the cumsum result (cumulative sum) of the group axis size. When groupListType is 1, the values in groupListOptional must be a non-negative sequence, representing the size of each group on the group axis. When groupListType is 2, the values in groupListOptional must be a non-negative sequence with the shape [E, 2], where E represents the group size, and the data layout is [[groupIdx0, groupSize0], [groupIdx1, groupSize1]...], where groupSize is the size of each group on the group axis. For details, see [groupListOptional Configuration Example](#grouplistoptional-configuration-example).

- groupType represents the axis to be grouped. For example, if matrix multiplication is C[m,n]=A[m,k]xB[k,n], then groupType value -1: no grouping, 0: grouping on the m axis, 2: grouping on the k axis. For details, see the <a href="#general-constraints">groupType supported scenarios</a> constraint.

- actType (int64_t, computation input): integer parameter, represents the activation function type. Value range: 0-5.

<a id="a8w8-scenario-constraints"></a>

<details>
<summary>A8W8 Scenario Constraints</summary>

- **Data Type Requirements**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | INT8 | INT8 (ND) | INT32/null | UINT64 | null | null | null | null | INT64 | null | null | null | INT8 |
  | INT8 | INT8 (ND/NZ) | INT32/null |BFLOAT16| null | null | null | FLOAT/null | INT64 | null | null | null | BFLOAT16|
  | INT8 | INT8 (NZ) | BFLOAT16/null |FLOAT/BFLOAT16| null | null | null | FLOAT/null | INT64 | null | null | null | BFLOAT16|
  | INT8 | INT8 (ND/NZ) | INT32/null | FLOAT | null | null | null | FLOAT/null | INT64 | null | null | null | FLOAT16 |
  | INT8 | INT8 (ND/NZ) | INT32/null | null | null | null | null | null | INT64 | null | null | null | INT32 |

- **Constraints**
  In addition to the [General Constraints](#general-constraints), the remaining constraints for the A8W8 scenario are as follows:

  - Only GroupType=0 (M-axis grouping) is supported.

  - Currently, only TensorLists with a length of 1 are supported for x, weight, and out.

  - x does not support transposition.

  - x only supports 2D tensors with shape (M, K).

  - weight only supports 3D tensors with shape (E, K, N) or (E, N, K).

  - To enable the fixed-axis algorithm for performance optimization, the following input shape and parameter configuration conditions must be met simultaneously:

    * Input shape conditions (any one of the following groups must be satisfied)

      The shape of x is (M, 7168), and the shape of weight is (7168, 4096).

      The shape of x is (M, 2048), and the shape of weight is (2048, 7168).

    * Parameter configuration conditions

      The first element of tuningConfigOptional: set to a value greater than 128 and less than 512.

      The second element of tuningConfigOptional: set to 0.

      The third element of tuningConfigOptional: set to -1, or set to a value greater than or equal to M × N × 4.

</details>

<a id="a8w4-scenario-constraints"></a>

<details>
<summary>A8W4 Scenario Constraints</summary>

- **Data Type Requirements**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | INT8 | INT4 (ND/NZ) | FLOAT | UINT64 | null | null | null | FLOAT | INT64 | null | null | null | BFLOAT16|
  | INT8 | INT4 (ND/NZ) | FLOAT | UINT64 | FLOAT/null | null | null | FLOAT | INT64 | null | null | null | FLOAT16 |

- **Constraints**

  In addition to the [General Constraints](#general-constraints), the remaining constraints for the A8W4 scenario are as follows:

  - Only GroupType=0 (M-axis grouping) and actType=0 are supported.
  - Currently, only TensorLists of length 1 are supported for x, weight, and out.
  - x does not support transposition, and weight does not support transposition.
  - x only supports 2D tensors with shape (M, K).
    - weight defaults to a 3D tensor with shape (E, K, N).
    - Bias is an auxiliary result computed offline during the computation process. Its value must be $8\times weight \times scale$, accumulated along the first dimension, and its shape must be $[E, N]$.
    - When the data type of weight is INT32, each INT32 is treated as eight INT4 values.
    - When offset is empty:
      - In this scenario, only groupListType 1 is supported (the operator does not check the value of groupListType and assumes groupListType is 1). k must be an integer multiple of quantGroupSize, and k <= 18432. quantGroupSize is the per-group quantization length in the k direction, and currently quantGroupSize=256 is supported.
      - In this scenario, n must be an integer multiple of 8.
      - In this scenario, scale is the result of offline fusion of pergroup and perchannel, and the shape must be $[E, quantGroupNum, N]$, where $quantGroupNum=k \div quantGroupSize$.
      - In this scenario, when the expected number of tokens processed by each expert is greater than n/4, i.e., when the first value in tuningConfigOptional is greater than n/4, better performance is usually achieved, and the memory usage increases by $g\times k \times n$ bytes (where g is the number of matmul groups, i.e., the number of groups).
    - When offset is not empty
      - scale is the result of offline fusion of pergroup and perchannel, and the shape must be $[E, 1, N]$.
      - In this scenario, offsetOptional is not empty. For asymmetric quantization, offsetOptional is the offline computation auxiliary result during the computation process, i.e., $antiquantOffset \times scale$, with a shape requirement of $[E, 1, N]$ and dtype of FLOAT32.
    - The second element of the tuningConfigOptional array can be set to 1 to enable a special format template for weight in the A8W4 scenario, so as to optimize operator performance (for the shape range with performance advantage, refer to: K >= 2048 && N >= 2048). Note that this template requires the weight shape to be (E, N, K), which is then converted via ND2NZ before being used as the operator input.
</details>

<a id="a16w4-scenario-constraints"></a>

<details>
<summary>A16W4 Scenario Constraints</summary>

- **Data Type Requirement**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | FLOAT16 | INT4 (ND) | FLOAT16/null | null | null | FLOAT16 | FLOAT16 | null | INT64 | null | null | null | FLOAT16 |
  | BFLOAT16| INT4 (ND) | FLOAT/null | null | null | BFLOAT16 | BFLOAT16 | null | INT64 | null | null | null | BFLOAT16|

- **Constraints**

  In addition to the [General Constraints](#general-constraints), the remaining constraints for the a16w4 scenario are as follows:

  - x does not support transposition.

  - Only GroupType=-1, 0, actType=0, and groupListType=0/1 are supported.

  - The last dimension of each tensor in weight must be an even number. The last dimension refers to the N axis of $weight_i$ when weight is not transposed, or the K axis of $weight_i$ when weight is transposed.

  - Symmetric quantization supports perchannel and pergroup quantization modes. For pergroup mode, the pergroup number G or $G_i$ must be divisible by the corresponding $k_i$.

  - Asymmetric quantization supports only the perchannel mode.

  - In the pergroup scenario, when weight is transposed, the pergroup length $s_i$ must be an even number.

  - If weight is multi-tensor, define the pergroup length $s_i = k_i / G_i$, and all $s_i(i=1,2,...g)$ must be equal.

  - The shape of pseudo-quantization parameters antiquantScaleOptional and antiquantOffsetOptional must satisfy the following table (where g is the number of matmul groups, G is the number of pergroups, and $G_i$ is the number of pergroups of the i-th tensor):

    | Use Scenario | Sub-Scenario | Shape Restriction |
    |:---------:|:-------:| :-------|
    | Pseudo-quantization perchannel | single weight | $[E, N]$|
    | Pseudo-quantization perchannel | multi-weight | $[N_i]$|
    | Pseudo-quantization pergroup | single weight | $[E, G, N]$|
    | Pseudo-quantization pergroup | multi-weight | $[G_i, N_i]$|

</details>

<a id="a16w8-scenario-constraints"></a>

<details>
<summary>A16W8 Scenario Constraints</summary>

- **Data Type Requirements**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | FLOAT16 | INT8 (ND) | FLOAT16/null | null | null | FLOAT16 | FLOAT16 | null | INT64 | null | null | null | FLOAT16 |
  | BFLOAT16| INT8 (ND) | FLOAT/null | null | null | BFLOAT16 | BFLOAT16 | null | INT64 | null | null | null | BFLOAT16|

- **Constraints**

  In addition to the [General Constraints](#general-constraints), the remaining constraints for the a16w8 scenario are as follows:

  - Transposition is not supported for x.

  - Only GroupType=-1, 0, actType=0, and groupListType=0/1 are supported.

  - Only per-channel quantization mode is supported.

  - If weight is a multi-tensor, define the per-group length $s_i = k_i / G_i$, and all $s_i(i=1,2,...g)$ must be equal.

  - The shapes of the pseudo-quantization parameters antiquantScaleOptional and antiquantOffsetOptional must satisfy the following table (where g is the number of matmul groups):

    | Use Scenario | Sub-Scenario | Shape Restriction |
    |:---------:|:-------:| :-------|
    | Pseudo-quantization per-channel | Single weight | $[E, N]$ |
    | Pseudo-quantization per-channel | Multiple weights | $[N_i]$ |

</details>

<a id="a4w4-scenario-constraints"></a>

<details>
<summary>A4W4 Scenario Constraints</summary>

- **Data Type Requirements**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | INT4 | INT4 (ND/NZ) | null | UINT64 | null | null | null | FLOAT/null | INT64 | null | null | null | FLOAT16/BFLOAT16|

- **Constraints**

  In addition to the [General Constraints](#general-constraints), the remaining constraints for the A4W4 scenario are as follows:

  - Only GroupType=0 (M-axis grouping), actType=0, and groupListType=0/1 are supported.

  - Currently, only TensorLists with a length of 1 are supported for x, weight, and out.

  - x does not support transposition, and weight does not support transposition.

  - x only supports a 2D tensor with the shape (M, K).

  - weight only supports a 3D tensor with shape (E, K, N).

  - When the data format of weight is ND, n must be an integer multiple of 8.

  - Per-channel and per-group quantization are supported. In the per-channel scenario, the shape of scale must be $[E, N]$; in the per-group scenario, it must be $[E, G, N]$.

  - In the per-group scenario, $G$ must be divisible by $k$, and $k/G$ must be an even number.

</details>

<a id="non-quantization-scenario-constraints"></a>

<details>
<summary>Non-Quantization Scenario Constraints</summary>

- **Data Type Requirements**

  | x | weight | bias | scale | offset | antiquantScale | antiquantOffset | perTokenScale | groupList | activationInput | activationQuantScale | activationQuantOffset | out |
  |---------|----------------|--------------|--------|------------|----------------|-----------------|---------------|-----------|-----------------|----------------------|-----------------------|---------|
  | FLOAT | FLOAT (ND) | FLOAT/null | null | null | null | null | null | INT64 | null | null | null | FLOAT |
  | FLOAT16 | FLOAT16 (ND/NZ) | FLOAT16/null | null | null | null | null | null | INT64 | null | null | null | FLOAT16 |
  | BFLOAT16| BFLOAT16(ND/NZ) | FLOAT/null | null | null | null | null | null | INT64 | null | null | null | BFLOAT16|

- **Constraints**

  In addition to the [General Constraints](#general-constraints), the remaining constraints for the non-quantization scenario are as follows:

  - Supports GroupType=-1, 0, 2, actType=0, groupListType=0/1.

</details>

<a id="groupType-constraints"></a>

<details>
<summary>groupType Supported Scenarios</summary>

- The a16w8 and a16w4 scenarios only support groupType of -1 and 0.

- In A8W8, A8W4, and A4W4 scenarios, only the groupType 0 scenario is supported, where the number of x tensors is single.

- The input types of x, weight, and y are aclTensorList, representing an array object of aclTensor type. In the following table, "single" indicates an aclTensorList consisting of one aclTensor, and "multiple" indicates an aclTensorList consisting of multiple aclTensors. For example, "single-multiple-single" means x is a single tensor, weight is multiple tensors, and y is a single tensor.

  | groupType | Number of x Tensors | Number of weight Tensors | Number of y Tensors | splitItem | groupListOptional | Transposition | Remaining Scenario Restrictions |
  |:---------:|:-------:|:-------:|:-------:|:--------:|:------------------|:--------| :-------|
  | -1 | Multiple | Multiple | Multiple | 0/1 | groupListOptional must be passed as null. | 1) Transposition of x is not supported.<br> 2) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent. | Tensors in x must have consistent dimensions and support 2D. Tensors in weight must be 2D. Tensors in y must have the same dimensions as x. |
  | 0 | Single | Single | Single | 2/3 | 1) groupListOptional must be passed.<br> 2) When groupListType is 0, the last value must be less than or equal to the first dimension of the tensor in x. When groupListType is 1, the sum of values must be less than or equal to the first dimension of the tensor in x. When groupListType is 2, the sum of values in the second column must be less than or equal to the first dimension of the tensor in x.<br> 3) The first dimension of groupListOptional supports a maximum of 1024, i.e., up to 1024 groups. | 1) Transposition of x is not supported.<br> 2) Weight transposition is supported. Weight transposition is not supported in A8W4 and A4W4 scenarios. | Tensors in weight must be 3D. Tensors in x and y must be 2D. |
  | 0 | Single | Multiple | Single | 2/3 | 1) groupListOptional must be passed.<br> 2) When groupListType is 0, the last value must be less than or equal to the first dimension of the tensor in x. When groupListType is 1, the sum of values must be less than or equal to the first dimension of the tensor in x. When groupListType is 2, the sum of values in the second column must be less than or equal to the first dimension of the tensor in x.<br> 3) The first dimension of groupListOptional supports a maximum of 128, i.e., up to 128 groups. | 1) Transposition of x is not supported.<br> 2) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent. | 1) Tensors in x, weight, and y must be 2D.<br> 2) The N axis of each tensor in weight must be equal. |
  | 0 | Multiple | Multiple | Single | 2/3 | 1) groupListOptional is optional.<br> 2) If groupListOptional is passed, when groupListType is 0, the differences of groupListOptional must correspond one-to-one with the first dimension of the tensors in x. When groupListType is 1, the values of groupListOptional must correspond one-to-one with the first dimension of the tensors in x. When groupListType is 2, the values in the second column of groupListOptional must correspond one-to-one with the first dimension of the tensors in x.<br> 3) The first dimension of groupListOptional supports a maximum of 128, i.e., up to 128 groups. | 1) Transposition of x is not supported.<br> 2) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent. | 1) Tensors in x, weight, and y must be 2D.<br> 2) The N axis of each tensor in weight must be equal. |
  | 2 | Single | Single | Single | 2/3 | 1) groupListOptional must be passed.<br> 2) When groupListType is 0, the last value must be less than or equal to the second dimension of the tensor in x. When groupListType is 1, the sum of values must be less than or equal to the second dimension of the tensor in x. When groupListType is 2, the sum of values in the second column must be less than or equal to the second dimension of the tensor in x.<br> 3) The first dimension of groupListOptional supports a maximum of 1024, i.e., up to 1024 groups. | 1) x must be transposed.<br> 2) Weight cannot be transposed. | 1) Tensors in x and weight must be 2D. Tensors in y must be 3D.<br> 2) bias must be passed as null. |
  | 2 | Single | Multiple | Multiple | 0/1 | groupListOptional must be passed as null. | 1) x must be transposed.<br> 2) Weight cannot be transposed. | 1) Tensors in x, weight, and y must be 2D.<br> 2) The weight length supports a maximum of 128, i.e., up to 128 groups.<br> 3) In the original shape, the sum of the first dimension of each tensor in weight must not exceed the first dimension of x.<br> 4) bias must be passed as null. |

</details>

<a id="grouplistoptional-configuration-example"></a>

<details>
<summary>groupListOptional configuration example</summary>

- shape information

  M = 789, K = 4096, N = 7168, E = 9 (experts 0, 2, and 5 have tokens to process; expert 0 processes 123 tokens, experts 2 and 5 process 333 tokens each)
  X shape: [[789, 4096]]
  W shape: [[9, 4096, 7168]]
  Y shape: [[789, 7168]]

- When groupListType is 0, groupList is configured as follows

  - groupListOptional: `[123, 123, 456, 456, 456, 789, 789, 789, 789]`

- When groupListType is 1, groupList is configured as follows:

  - groupListOptional: `[123, 0, 333, 0, 0, 333, 0, 0, 0]`

- When groupListType is 2, groupList is configured as follows:

  - In this mode, groupListOptional moves all non-zero groups to the front, which is suitable for scenarios with many inactive experts.

  - groupListOptional: `[[0, 123], [2, 333], [5, 333], [1, 0], [3, 0], [4, 0], [6, 0], [7, 0], [8, 0]]`

</details>

Ascend 950PR/Ascend 950DT

  • Common constraints:

    • groupListType: supports values 0 and 1. When groupListType is 0, groupListOptional must be a non-negative monotonically non-decreasing sequence. When groupListType is 1, groupListOptional must be a non-negative sequence.

    • actType (int64_t, computation input): integer parameter that represents the activation function type. The value range is 0–5.

      • In pseudo-quantization and non-quantization scenarios, actType supports only 0.

      • In the full quantization scenario, when x and weight are INT8, the quantization mode is static T-C quantization or dynamic K-C quantization, and the scale data type is FLOAT32 or BFLOAT16, actType supports 0, 1, 2, 4, and 5. In the remaining full quantization scenarios, actType supports only 0.

    Static Quantization Scenario Constraints

    • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, perTokenScaleOptional, activationInputOptional

    • The supported data type combinations for non-empty parameters must satisfy the following table:

      groupTypexweightbiasOptionalscaleOptionalout
      0INT8INT8INT32/nullUINT64/INT64BFLOAT16/FLOAT16/INT8
      0INT8INT8INT32/nullnull/UINT64/INT64INT32
      0INT8INT8INT32/BFLOAT16/FLOAT32/nullBFLOAT16/FLOAT32BFLOAT16
      0INT8INT8INT32/FLOAT16/FLOAT32/nullFLOAT32FLOAT16
      0HIFLOAT8HIFLOAT8nullUINT64/INT64BFLOAT16/FLOAT16/ FLOAT32
      0/2HIFLOAT8HIFLOAT8nullFLOAT32BFLOAT16/FLOAT16/FLOAT32
      0FLOAT8_E5M2/FLOAT8_E4M3FNFLOAT8_E5M2/FLOAT8_E4M3FNnullUINT64/INT64BFLOAT16/FLOAT16/FLOAT32
      0/2FLOAT8_E5M2/FLOAT8_E4M3FNFLOAT8_E5M2/FLOAT8_E4M3FNnullFLOAT32BFLOAT16/FLOAT16/FLOAT32

    • scaleOptional must satisfy the following table (where g is the number of matmul groups, i.e., the number of groups):

      groupTypeuse scenarioshape restriction
      0/2weight single tensorperchannel scenario: each tensor is 2-dimensional, with shape (g, N); pertensor scenario: each tensor is 2-dimensional or 1-dimensional, with shape (g, 1) or (g,)
    Dynamic Quantization (T-T && T-C && K-T && K-C Quantization) Scenario Constraints

    • The supported input types for the dynamic quantization (T-T && T-C && K-T && K-C quantization) scenario are:

      • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, activationInputOptional

      • The supported data type combinations for non-empty parameters must satisfy the following table:

        groupTypexweightbiasOptionalscaleOptionalperTokenScaleOptionalout
        0INT8INT8INT32/BFLOAT16/FLOAT32/nullBFLOAT16/FLOAT32FLOAT32BFLOAT16
        0INT8INT8INT32/FLOAT16/FLOAT32/nullFLOAT32FLOAT32FLOAT16
        0/2HIFLOAT8HIFLOAT8nullFLOAT32FLOAT32BFLOAT16/ FLOAT16/FLOAT32
        0/2FLOAT8_E5M2/FLOAT8_E4M3FNFLOAT8_E5M2/FLOAT8_E4M3FNnullFLOAT32FLOAT32BFLOAT16/ FLOAT16/FLOAT32

      • scaleOptional must satisfy the following table (where g is the number of groups i.e. split), and it is recommended to use the shape (g,) for scaleOptional in the pertensor scenario to avoid confusion with the G-B quantization mode:

        groupTypeuse scenarioshape restriction
        0/2weight single tensorperchannel scenario: each tensor is 2-dimensional, with shape (g, N); pertensor scenario: each tensor is 2-dimensional or 1-dimensional, with shape (g, 1) or (g,)

      • perTokenScaleOptional must satisfy the following table:

        groupTypeuse scenarioshape restriction
        0x single tensorpertoken scenario: each tensor is 1-dimensional, with shape (M,); pertensor scenario: each tensor is 2-dimensional or 1-dimensional, with shape (g, 1) or (g,). The pertensor scenario is not supported when the input is INT8.
        2x single tensorpertoken scenario: each tensor is 2-dimensional, with shape (g, M); pertensor scenario: each tensor is 2-dimensional or 1-dimensional, with shape (g, 1) or (g,)
    Dynamic Quantization (MX Quantization) Scenario Constraints

    • The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, activationInputOptional.

    • In the formula, the quantization block sizes are: gsM = gsN = 1, gsK = 32. MX quantization is a special per-group quantization.

    • The supported data type combinations for non-empty parameters must satisfy the following table:

      groupTypexweightbiasOptionalscaleOptionalperTokenScaleOptionalout
      0/2FLOAT8_E5M2/FLOAT8_E4M3FNFLOAT8_E5M2/FLOAT8_E4M3FNnullFLOAT8_E8M0FLOAT8_E8M0BFLOAT16/FLOAT16/FLOAT32
      0FLOAT4_E2M1FLOAT4_E2M1FLOAT32/nullFLOAT8_E8M0FLOAT8_E8M0BFLOAT16/FLOAT16/FLOAT32

    • scaleOptional must satisfy the following table (where g is the number of matmul groups i.e. split, and g_i is the i-th group (subscript starting from 0)):

      groupTypeUse ScenarioShape Restriction
      0weight single tensorEach tensor is 4-dimensional. When weight is transposed, the shape is (g, N, ceil(K / 64), 2); when weight is not transposed, the shape is (g, ceil(K / 64), N, 2).
      2weight single tensorEach tensor is 3-dimensional, with shape ((K / 64) + g, N, 2). The start address offset of scale_i is ((K_0 + K_1 + ...+ K_{i-1})/ 64 + g_i) * N * 2, i.e., the start address offset of scale_0 is 0, the start address offset of scale_1 is (K_0 / 64 + 1) * N * 2, the start address offset of scale_2 is ((K_0 + K_1) / 64 + 2) * N * 2, and so on.

    • perTokenScaleOptional must satisfy the following table:

      groupTypeUse ScenarioShape Restriction
      0x single tensorEach tensor is 3-dimensional, with shape (M, ceil(K / 64), 2).
      2x single tensorEach tensor is 3-dimensional, with shape ((K / 64) + g, M, 2). The start address offset is the same as scale.

    • For mx quantization where the input x is FLOAT4_E2M1, K must be an even number and not equal to 2. When weight is not transposed, N must also be an even number.

    Dynamic Quantization (G-B Quantization) Scenario Constraint

    • The data types supported in the dynamic quantization (G-B quantization) scenario are:

    • The following input parameters are empty: biasOptional, offsetOptional, antiQuantScaleOptional, antiQuantOffsetOptional, activationInputOptional

    • Formula: quantization block size: currently only gsM = 1, gsN = gsK = 128 are supported.

    • The supported data type combinations for non-empty parameters must satisfy the following table:

      groupTypexweightscaleOptionalperTokenScaleOptionalout
      0/2HIFLOAT8HIFLOAT8FLOAT32FLOAT32BFLOAT16/FLOAT16/ FLOAT32
      0/2FLOAT8_E5M2/FLOAT8_E4M3FNFLOAT8_E5M2/FLOAT8_E4M3FNFLOAT32FLOAT32BFLOAT16/FLOAT16/FLOAT32

    • scaleOptional must satisfy the following table (where g is the number of matmul groups, i.e., the number of groups, and g_i is the i-th group (subscript starting from 0)):

      groupTypeuse scenarioshape restriction
      0weight single tensorEach tensor is 3-dimensional. When weight is transposed, the shape is (g, ceil(N / gsN), ceil(K / gsK)); when weight is not transposed, the shape is (g, ceil(K / gsK), ceil(N / gsN)).
      2weight single tensorEach tensor is 2-dimensional, with shape (K / gsK + g, ceil(N / gsN)). The address offset of scale_i is ((K_0 + K_1 + ... + K_{i-1}) / gsK + g_i) ceil(N / gsN), i.e., the starting address offset of scale_0 is 0, the starting address offset of scale_1 is (K_0 / gsK + 1) ceil(N / gsN), the starting address offset of scale_2 is ((K_0 + K_1) / gsK + 2) * ceil(N / gsN), and so on.

    • perTokenScaleOptional must satisfy the following table:

      groupTypeUse ScenarioShape Restriction
      0x single tensorEach tensor is 2-dimensional with shape (M, ceil(K / gsK)).
      2x single tensorEach tensor is 2-dimensional with shape (K / gsK + g, M). The address offset of per_token_scale_i is ((K_0 + K_1 + ...+ K_{i-1}) / gsK + g_i) * M, i.e., the starting address offset of per_token_scale_0 is 0, the starting address offset of per_token_scale_1 is (K_0 / gsK + 1) * M, the starting address offset of per_token_scale_2 is ((K_0 + K_1) / gsK + 2) * M, and so on.

    • Special handling for dynamic quantization scenarios:

      • In the dynamic quantization scenario with M grouping or K grouping, when N equals 1 and the shape of scaleOptional is (g, 1), if weight can use either pertensor quantization or perchannel quantization, pertensor quantization mode is preferred.

      • In the dynamic quantization scenario with M grouping, when g = M and the shape of perTokenScaleOptional is (g,), x uses the pertoken quantization mode; when g = M, K <= 128, and the shape of perTokenScaleOptional is (g, 1), the quantization mode of x is selected based on the quantization mode of weight (if weight uses perchannel or pertensor quantization, x uses pertensor quantization; if weight uses perblock quantization, x uses pergroup quantization).

      • In the dynamic quantization scenario with K grouping, when K is less than 128, N is less than or equal to 128, and the shape of scaleOptional is (g, 1), according to the existing quantization mode differentiation rules, it can be either non-pergroup quantization or G-B quantization. In this scenario, it is uniformly treated as G-B quantization.

      • In the dynamic quantization scenario with K grouping, when M equals 1 and the shape of perTokenScaleOptional is (g, 1), if x can be either pertoken quantized or pertensor quantized, the pertensor quantization mode is preferentially selected.

      • In the dynamic quantization scenario with K grouping, when K is less than 128, M equals 1, and the shape of perTokenScaleOptional is (g, 1), if N is less than or equal to 128, x selects pergroup quantization; if N is greater than 128, the quantization mode of x is selected based on the quantization mode of weight (if weight uses perchannel or pertensor quantization, x selects pertensor quantization; if weight uses perblock quantization, x selects pergroup quantization).

      • In the dynamic quantization scenario with K grouping, when K is less than 128 and M is not equal to 1, if N is less than or equal to 128, x selects pergroup quantization; if N is greater than 128, the quantization mode of x is selected based on the quantization mode of weight (if weight uses perchannel or pertensor quantization, x selects pertoken quantization; if weight uses perblock quantization, x selects pergroup quantization).

    Non-Quantization Scenario Constraints

    • The data types supported in the non-quantization scenario are:

      • The following input parameters are empty: scaleOptional, offsetOptional, antiQuantScaleOptional, antiQuantOffsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional, activationFeatureOutOptional

      • The data type combinations supported for non-empty parameters must satisfy the following table

        groupTypexweightbiasOptionalout
        -1/0/2BFLOAT16BFLOAT16BFLOAT16/FLOAT32/nullBFLOAT16
        -1/0/2FLOAT16FLOAT16FLOAT16/FLOAT32/nullFLOAT16
        -1/0/2FLOAT32FLOAT32FLOAT32/nullFLOAT32
    Pseudo-Quantization Scenario Constraint

    • The data types supported in the pseudo-quantization scenario are:

      • The following input parameters are empty: scaleOptional, offsetOptional, perTokenScaleOptional, activationInputOptional, activationQuantScaleOptional, activationQuantOffsetOptional

      • The data type combinations supported by non-empty parameters must satisfy the following table

        groupTypexweightbiasOptionalantiquantScaleOptionalantiquantOffsetOptionalout
        -1/0BFLOAT16INT8/INT4BFLOAT16/FLOAT32/nullBFLOAT16BFLOAT16/ nullBFLOAT16
        -1/0FLOAT16INT8/INT4FLOAT16/nullFLOAT16FLOAT16/nullFLOAT16
        0BFLOAT16FLOAT8_E5M2/FLOAT8_E4M3FN/HIFLOAT8BFLOAT16/FLOAT32/ nullBFLOAT16nullBFLOAT16
        0FLOAT16FLOAT8_E5M2/FLOAT8_E4M3FN/HIFLOAT8FLOAT16/nullFLOAT16nullFLOAT16

      • When the data type of weight is FLOAT8_E5M2, FLOAT8_E4M3FN, or HIFLOAT8, antiquantOffsetOptional only supports passing a null pointer or an empty tensorList, and weight only supports transposition.

      • If the type of weight is INT4, the last dimension of each tensor in weight must be an even number. The last dimension of weightiweight_i refers to the N axis of weightiweight_i when weight is not transposed, or the K axis of weightiweight_i when weight is transposed.

      • antiquantScaleOptional and non-null biasOptional and antiquantOffsetOptional must satisfy the following table (where g is the number of matmul groups, i.e., the number of groups):

        groupTypeUse ScenarioShape Restriction
        -1Weight multi-tensorEach tensor is 1D with shape (nin_i). It is not allowed that some tensors in a tensorList have shape (nin_i) while others are null.
        0Weight single-tensorEach tensor is 2D with shape (g, N).
    Constraints for Different groupType Values

    • Supported scenarios for different groupType values:

      • In supported scenarios, "single" indicates a single tensor and "multiple" indicates multiple tensors. The order is x, weight, out. For example, "single-multiple-single" indicates that x is a single tensor, weight consists of multiple tensors, and out is a single tensor.

        groupTypeSupported ScenarioScenario Restriction
        -1multiple-multiple-multiple1) Only supports splitItem being 0 or 1.
        2) In the non-quantization scenario, each tensor in x and out must be 2-dimensional, with shapes of (mim_i, kik_i) and (mim_i, nin_i) respectively. In the pseudo-quantization scenario, tensors in x must have consistent dimensions, supporting 2 to 6 dimensions, and tensors in y must have the same dimensions as x. Each tensor in weight must be 2-dimensional, with a shape of (nin_i, kik_i) or (kik_i, nin_i). Each tensor in bias must be 1-dimensional, with a shape of (nin_i).
        3) groupListOptional must be passed as null.
        4) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent.
        5) x does not support transposition.
        6) Only non-quantization and pseudo-quantization scenarios are supported.
        7) Only ND in, ND out is supported.
        0single-single-single1) Only supports splitItem being 2 or 3.
        2) Each tensor in weight must be 3-dimensional, with a shape of (g, N, K) or (g, K, N). Each tensor in x and out must be 2-dimensional, with shapes of (M, K) and (M, N) respectively. Each tensor in bias must be 2-dimensional, with a shape of (g, N).
        3) groupListOptional must be passed. When groupListType is 0, the last value must not exceed the first dimension of the tensor in x. When groupListType is 1, the sum of values must not exceed the first dimension of the tensor in x.
        4) The first dimension of groupListOptional supports a maximum of 1024, i.e., up to 1024 groups.
        5) Supports x without transposition, and both transposed and non-transposed weight.
        6) Only ND in, ND out is supported.
        0single-multiple-single1) Only supports splitItem being 2 or 3.
        2) groupListOptional must be passed. When groupListType is 0, the last value must equal the first dimension of the tensor in x. When groupListType is 1, the sum of values must equal the first dimension of the tensor in x, with a maximum length of 1024.
        3) Each tensor in x and out must be 2-dimensional, with shapes of (M, K) and (M, N) respectively. Each tensor in weight must be 2-dimensional, with a shape of (N, K) or (K, N). Each tensor in bias must be 1-dimensional, with a shape of (N).
        4) The N axis of each tensor in weight must be equal.
        5) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent.
        6) x does not support transposition.
        7) Only non-quantization is supported.
        8) Only ND in, ND out is supported.
        0multiple-multiple-single1) Only supports splitItem being 2.
        2) Each tensor in x and out must be 2-dimensional, with shapes of (M, K) and (M, N) respectively. Each tensor in weight must be 2-dimensional, with a shape of (N, K) or (K, N). Each tensor in bias must be 1-dimensional, with a shape of (N).
        3) The N axis of each tensor in weight must be equal.
        4) If groupListOptional is passed, when groupListType is 0, the differences of groupListOptional must correspond one-to-one with the first dimension of each tensor in x. When groupListType is 1, the values of groupListOptional must correspond one-to-one with the first dimension of each tensor in x, with a maximum length of 1024.
        5) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must remain consistent.
        6) x does not support transposition.
        7) Only non-quantization is supported.
        8) Only ND in, ND out is supported.
        2single-single-single1) Only supports splitItem being 2 or 3.
        2) Each tensor in x and weight must be 2-dimensional, with shapes of (K, M) and (K, N) respectively. Each tensor in out must be 3-dimensional, with a shape of (g, M, N).
        3) groupListOptional must be passed. When groupListType is 0, the last value must not exceed the first dimension of the tensor in x. When groupListType is 1, the sum of values must not exceed the first dimension of the tensor in x.
        4) The first dimension of groupListOptional supports a maximum of 1024, i.e., up to 1024 groups.
        5) x must be transposed, and weight cannot be transposed.
        6) Only non-quantization and quantization scenarios are supported.
        7) Only ND in, ND out is supported.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul_v4.h"

#define CHECK_RET(cond, return_expr) \
  do {                               \
    if (!(cond)) {                   \
      return_expr;                   \
    }                                \
  } while (0)

#define LOG_PRINT(message, ...)     \
  do {                              \
    printf(message, ##__VA_ARGS__); \
  } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
      shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
  // Boilerplate: resource initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor_New(const std::vector<int64_t>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                      aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to apply for device-side memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

  // Copy host data to the device memory by calling aclrtMemcpy.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Compute the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
      strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Create an aclTensor by calling aclCreateTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<int64_t>& shape, void** deviceAddr,
                  aclDataType dataType, aclTensor** tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Allocate device memory by calling aclrtMalloc.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

  // Copy host data to the device memory by calling aclrtMemcpy.
  std::vector<T> hostData(size, 0);
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
      strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call the aclCreateTensor API to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}


int CreateAclTensorList(const std::vector<std::vector<int64_t>>& shapes, void** deviceAddr,
                      aclDataType dataType, aclTensorList** tensor) {
  int size = shapes.size();
  std::vector<aclTensor*> tensors(size);
  for (int i = 0; i < size; i++) {
      int ret = CreateAclTensor<uint16_t>(shapes[i], deviceAddr + i, dataType, tensors.data() + i);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
  }
  *tensor = aclCreateTensorList(tensors.data(), size);
  return ACL_SUCCESS;
}


int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
  // Set deviceId based on the actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  // Handle the check based on your requirements.
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the input and output. Customize the construction based on the API interface.
  std::vector<std::vector<int64_t>> xShape = {{512, 256}};
  std::vector<std::vector<int64_t>> weightShape= {{2, 256, 256}};
  std::vector<std::vector<int64_t>> biasShape = {{2, 256}};
  std::vector<std::vector<int64_t>> yShape = {{512, 256}};
  std::vector<int64_t> groupListShape = {{2}};
  std::vector<int64_t> groupListData = {256, 512};
  void* xDeviceAddr[1];
  void* weightDeviceAddr[1];
  void* biasDeviceAddr[1];
  void* yDeviceAddr[1];
  void* groupListDeviceAddr;
  aclTensorList* x = nullptr;
  aclTensorList* weight = nullptr;
  aclTensorList* bias = nullptr;
  aclTensor* groupedList = nullptr;
  aclTensorList* scale = nullptr;
  aclTensorList* offset = nullptr;
  aclTensorList* antiquantScale = nullptr;
  aclTensorList* antiquantOffset = nullptr;
  aclTensorList* perTokenScale = nullptr;
  aclTensorList* activationInput = nullptr;
  aclTensorList* activationQuantScale = nullptr;
  aclTensorList* activationQuantOffset = nullptr;
  aclTensorList* out = nullptr;
  aclTensorList* activationFeatureOut = nullptr;
  aclTensorList* dynQuantScaleOut = nullptr;
  int64_t splitItem = 3;
  int64_t groupType = 0;
  int64_t groupListType = 0;
  int64_t actType = 0;

  // Create the x aclTensorList.
  ret = CreateAclTensorList(xShape, xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the weight aclTensorList.
  ret = CreateAclTensorList(weightShape, weightDeviceAddr, aclDataType::ACL_FLOAT16, &weight);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the bias aclTensorList.
  ret = CreateAclTensorList(biasShape, biasDeviceAddr, aclDataType::ACL_FLOAT16, &bias);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the y aclTensorList.
  ret = CreateAclTensorList(yShape, yDeviceAddr, aclDataType::ACL_FLOAT16, &out);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  // Create the group_list aclTensor.
  ret = CreateAclTensor_New<int64_t>(groupListData, groupListShape, &groupListDeviceAddr, aclDataType::ACL_INT64, &groupedList);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  uint64_t workspaceSize = 0;
  aclOpExecutor* executor;

  // 3. Call the CANN operator library API.
  // Call the first-phase API of aclnnGroupedMatmulV4.
  ret = aclnnGroupedMatmulV4GetWorkspaceSize(x, weight, bias, scale, offset, antiquantScale, antiquantOffset, perTokenScale, groupedList, activationInput, activationQuantScale, activationQuantOffset, splitItem, groupType, groupListType, actType, out, activationFeatureOut, dynQuantScaleOut, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
  // Apply for device memory based on the workspaceSize calculated by the first-phase API.
  void* workspaceAddr = nullptr;
  if (workspaceSize > 0) {
      ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
  }
  // Call the second-phase API of aclnnGroupedMatmulV4.
  ret = aclnnGroupedMatmulV4(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmul failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronously wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the Device memory to the Host memory. Modify based on the specific API definition.
  for (int i = 0; i < 1; i++) {
      auto size = GetShapeSize(yShape[i]);
      std::vector<uint16_t> resultData(size, 0);
      ret = aclrtMemcpy(resultData.data(), size * sizeof(resultData[0]), yDeviceAddr[i],
                        size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
      for (int64_t j = 0; j < size; j++) {
          LOG_PRINT("result[%ld] is: %d\n", j, resultData[j]);
      }
  }

  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensorList(x);
  aclDestroyTensorList(weight);
  aclDestroyTensorList(bias);
  aclDestroyTensorList(out);

  // 7. Release Device resources. Modify based on the specific API definition.
  for (int i = 0; i < 1; i++) {
      aclrtFree(xDeviceAddr[i]);
      aclrtFree(weightDeviceAddr[i]);
      aclrtFree(biasDeviceAddr[i]);
      aclrtFree(yDeviceAddr[i]);
  }
  if (workspaceSize > 0) {
      aclrtFree(workspaceAddr);
  }
  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details