aclnnGroupedMatmulSwigluQuantV2

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Fuses GroupedMatmul, dequant, swiglu, and quant. For details, see Formula.

Compared with the aclnnGroupedMatmulSwigluQuant API, this API adds:
- Ascend 950PR/Ascend 950DT:
  - Added MXFP8, MXFP4, and Pertoken quantization scenarios.
  - The field types of the parameters weight, weightScale, and weightAssistMatrix are changed to tensorlist. Select the appropriate API based on the actual situation.
Formula:
- x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.
  Quantization scenario A8W8 (A refers to the activation matrix, W refers to the weight matrix, and 8 refers to the INT8 data type):
  - Definition:
    - ⋅ denotes matrix multiplication.
    - ⊙ denotes element-wise multiplication.
    - $\left \lfloor x\right \rceil$ denotes rounding x to the nearest integer.
    - $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
    - $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
  - Input:
    - $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
    - $W∈\mathbb{Z_8}^{E \times K \times N}$ : group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
    - $w\_scale∈\mathbb{R}^{E \times N}$ : per-channel scale factor of the group weight matrix (right matrix), where E is the number of experts and N is the output dimension.
    - $x\_scale∈\mathbb{R}^{M}$ : per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.
    - $grouplist∈\mathbb{N}^{E}$ : Group index list of cumsum or count.
  - Output:
    - $Q∈\mathbb{Z_8}^{M \times N / 2}$ : Quantized output matrix.
    - $Q\_scale∈\mathbb{R}^{M}$ : Quantization scale factor.
  - Computation process
    - Determine the tokens of the current group based on groupList[i], where $i \in [0,Len(groupList)]$ .
      
      Example: Assume groupList=[3,4,4,6], groupListType=cumsum, or groupList=[3,1,0,2], groupListType=count.
      
      Note: The two different grouping methods above actually produce the same grouping result.
      
      The 0th right matrix W[0,:,:] corresponds to tokens x[0:3] at index positions [0,3) (3-0=3 tokens in total), corresponding to x_scale[0:3], w_scale[0], bias[0], offset[0], Q[0:3], Q_scale[0:3], Q_offset[0:3];
      
      The 1st right matrix W[1,:,:] corresponds to tokens x[3:4] at index positions [3,4) (4-3=1 token in total), corresponding to x_scale[3:4], w_scale[1], bias[1], offset[1], Q[3:4], Q_scale[3:4], Q_offset[3:4];
      
      The 2nd right matrix W[2,:,:] corresponds to tokens x[4:4] at index positions [4,4) (4-4=0 tokens in total), corresponding to x_scale[4:4], w_scale[2], bias[2], offset[2], Q[4:4], Q_scale[4:4], Q_offset[4:4];
      
      The 3rd right matrix W[3,:,:] corresponds to tokens x[4:6] at index positions [4,6) (6-4=2 tokens in total), corresponding to x_scale[4:6], w_scale[3], bias[3], offset[3], Q[4:6], Q_scale[4:6], Q_offset[4:6];
      
      Note: Parts not specified in grouplist will not participate in the update. For example, when groupList=[12,14,18], GroupListType=cumsum, and the shape of X is [30,:].
      
      Then the shape of the first output Q is [30,:], where the part Q[18:,:] will not be updated or initialized, and the data therein is the original data at the time of memory space allocation.
      
      Similarly, the shape of the second output Q is [30], where the part Q_scale[18:] will not be updated or initialized, and the data therein is the original data at the time of memory space allocation.
      
      That is, Q[:grouplist[-1],:] and Q_scale[:grouplist[-1]] of the output are the valid data parts.
    - Perform the following computation based on the input parameters determined by the grouping:
      
      $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$
      
      $C_{i,act}, gate_{i} = split(C_{i})$
      
      $S_{i}=Swish(C_{i,act})\odot gate_{i}$ where $Swish(x)=\frac{x}{1+e^{-x}}$
    - Quantize the output result
      
      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$
      
      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

MSD scenario A8W4 (A refers to the activation matrix, W refers to the weight matrix, 8 refers to the INT8 data type, 4 refers to the INT4 data type):

Definition:
- ⋅ denotes matrix multiplication.
- ⊙ denotes element-wise multiplication.
- $\left \lfloor x\right \rceil$ denotes rounding $x$ to the nearest integer.
- $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
- $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
- $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
Input:
- $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
- $W∈\mathbb{Z_4}^{E \times K \times N}$ : group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
- $weightAssistMatrix∈\mathbb{R}^{E \times N}$ : Auxiliary matrix for matrix multiplication computation (see the computation process for generating the auxiliary matrix below).
- $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$ : Per-channel scale factor of the group weight matrix (right matrix), where E is the number of experts, K_group_num is the number of groups along the K axis, and N is the output dimension.
- $x\_scale∈\mathbb{R}^{M}$ : Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.
- $grouplist∈\mathbb{N}^{E}$ : Group index list of cumsum or count.
Output:
- $Q∈\mathbb{Z_8}^{M \times N / 2}$ : quantized output matrix.
- $Q\_scale∈\mathbb{R}^{M}$ : quantization scale factor.
Computation Process
- 1. Determine the tokens of the current group based on groupList[i], $i \in [0,Len(groupList)]$ .
  - The grouping logic is the same as that of A8W8.
- 1. The computation process for generating the assist matrix (weightAssistMatrix) (note that the weightAssistMatrix is partially computed offline and provided as input, rather than being completed within the operator):
  - For per-channel quantization ( $w\_scale$ is 2-dimensional):
    
    $weightAssistMatrix_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$
  - For per-group quantization ( $w\_scale$ is 3-dimensional):
    
    $weightAssistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$
    
    NOTE

$num\_per\_group = K // K\_group\_num$

    - 3. Perform the following computation based on the input parameters determined by the group:

      - 3.1. Convert the left matrix $\mathbb{Z_8}$ into high and low $\mathbb{Z_4}$ parts:

        $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$
        $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$

      - 3.2. Enable per-channel or per-group quantization during matrix multiplication.

        per-channel:

        $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

        $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$

        per-group:

        $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k *   num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

        $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k *   num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$

      - 3.3. Combine the high-bit and low-bit matrix multiplication results into the overall result.

        $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAssistMatrix_{i}) \odot x\_scale_{i}$

        $C_{i,act}, gate_{i} = split(C_{i})$

        $S_{i}=Swish(C_{i,act})\odot gate_{i}$     where $Swish(x)=\frac{x}{1+e^{-x}}$

    - 3. Quantize the output result.

      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$
</details>

<details>
<summary>Quantization scenario A4W4 (A refers to the activation matrix, W refers to the weight matrix, and 4 refers to the INT4 data type):</summary>
<a id="quantization-scenario-a4w4"></a>

  - **Definition**:

    * **⋅** denotes matrix multiplication.

    * **⊙** denotes element-wise multiplication.

    * $\left \lfloor x\right \rceil$ denotes rounding $x$ to the nearest integer.

    * $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$

    * $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$

    * $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$

  - **Input**:

    * $X∈\mathbb{Z_4}^{M \times K}$: activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.

    * $W∈\mathbb{Z_4}^{E \times K \times N}$: group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.

    * $w\_scale∈\mathbb{R}^{E \times N}$: per-channel scale factor of the group weight matrix (right matrix), where E is the number of experts and N is the output dimension.

    * $x\_scale∈\mathbb{R}^{M}$: Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.

    * $smoothScale∈\mathbb{R}^{E \times N/2}(per-channel) or \mathbb{R}^{E}(per-tensor)$: Smooth scale factor, where E is the number of experts and N is the output dimension.

    * $grouplist∈\mathbb{N}^{E}$: Group index list of cumsum or count.

  - **Output**:

    * $Q∈\mathbb{Z_8}^{M \times N / 2}$: Quantized output matrix.

    * $Q\_scale∈\mathbb{R}^{M}$: Quantization scale factor.

  - **Computation process**

    - 1. Determine the tokens of the current group based on groupList[i], where $i \in [0,Len(groupList)]$.

      - The grouping logic is the same as that of A8W8.

    - 2. Perform the following computation based on the input parameters determined by the grouping:

      $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$

      $C_{i,act}, gate_{i} = split(C_{i})$

      $S_{i}=Swish(C_{i,act})\odot gate_{i}$   where $Swish(x)=\frac{x}{1+e^{-x}}$

      $S_{i} = S_{i} \odot smoothScale_{i\ Broadcast}$

    - 3. Quantize the output result.

      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$

      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$
</details>

Ascend 950PR/Ascend 950DT:
MX Quantization Scenario:
- Definition:
  - ⋅ denotes matrix multiplication.
  - ⊙ denotes element-wise multiplication.
- Input:
  - $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
  - $W∈\mathbb{Z_8}^{E \times K \times N}$ : group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
  - $w\_scale∈\mathbb{R}^{E \times ceil(K / 64) \times N \times 2}$ : Per-channel scale factor of the group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
  - $x\_scale∈\mathbb{R}^{M \times ceil(K / 64) \times 2}$ : Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
  - $grouplist∈\mathbb{N}^{E}$ : Group index list of cumsum or count.
- Output:
  - $Q∈\mathbb{Z_8}^{M \times N / 2}$ : Quantized output matrix.
  - $Q\_scale∈\mathbb{R}^{M \times ceil((N / 2) / 64) \times 2}$ : Quantization scale factor.
- Computation Process
  - 1. Determine the tokens of the current group based on groupList[i], where $i \in [0,Len(groupList)]$ .
  - 1. Perform the following computation based on the input parameters determined for the group:
    $C_{i} = (X_{i}\cdot W_{i} )\odot xScale_{i\ Broadcast} \odot wScale_{i\ Broadcast}$
    
    $C_{i,act}, gate_{i} = split(C_{i})$
    
    $S_{i}=Swish(C_{i,act})\odot gate_{i}$ , where $Swish(x)=\frac{x}{1+e^{-x}}$
  - 1. Quantize the output result.
    $shared\_exp = \left\lfloor \log_2(max_i(|S_i|)) \right\rceil - emax$
    
    $QScale = 2 ^ {shared\_exp}$
    
    $Q_i = quantize\_to\_element\_format(S_i/Qscale), \space i\space from\space 1\space to\space blocksize$
    - $emax$ : The exponent of the maximum normal number of the corresponding data type.
      
      DataType emax
      FLOAT8_E4M3FN 8
      FLOAT8_E5M2 15
      FLOAT4_E2M1 2
    - $blocksize$ : The number of elements quantized each time. Only 32 is supported.
Pertoken quantization scenario:
- Definition:
  - ⋅ denotes matrix multiplication.
  - ⊙ denotes element-wise multiplication.
- Input:
  - $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
  - $W∈\mathbb{Z_8}^{E \times K \times N}$ : group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
  - $w\_scale∈\mathbb{R}^{E \times N}$ : per-channel scale factor of the group weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
  - $x\_scale∈\mathbb{R}^{M}$ : per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
  - $grouplist∈\mathbb{N}^{E}$ : group index list of cumsum or count.
- Output:
  - $Q∈\mathbb{Z_8}^{M \times N / 2}$ : quantized output matrix.
  - $Q\_scale∈\mathbb{R}^{M}$ : quantization scale factor.
- Computation Process
  - 1. Determine the tokens of the current group based on groupList[i], $i \in [0,Len(groupList)]$
    - Perform the following computation based on the input parameters determined by the group:
      
      $C_{i} = (X_{i}\cdot W_{i} )\odot xScale_{i} \odot wScale_{i}$
      
      $C_{i,act}, gate_{i} = split(C_{i})$
      
      $S_{i}=Swish(C_{i,act})\odot gate_{i}$ , where $Swish(x)=\frac{x}{1+e^{-x}}$
      
      where $xScale_{i}$ represents the quantization scale factor corresponding to the token.
    - Quantized output
      
      $Q\_scale_{i} = \frac{max(|S_{i}|)}{max(type)}$
      
      $Q_{i} = \lfloor \frac{S_{i}}{Q\_scale_{i}} \rceil$

DataType	emax
FLOAT8_E4M3FN	8
FLOAT8_E5M2	15
FLOAT4_E2M1	2

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation flow, and then call the "aclnnGroupedMatmulSwigluQuantV2" API to perform the computation.

aclnnStatus aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize(
    const aclTensor     *x,
    const aclTensorList *weight,
    const aclTensorList *weightScale,
    const aclTensorList *weightAssistMatrix,
    const aclTensor     *bias,
    const aclTensor     *xScale,
    const aclTensor     *smoothScale,
    const aclTensor     *groupList,
    int64_t              dequantMode,
    int64_t              dequantDtype,
    int64_t              quantMode,
    int64_t              groupListType,
    const aclIntArray   *tuningConfig,
    aclTensor           *output,
    aclTensor           *outputScale,
    uint64_t            *workspaceSize,
    aclOpExecutor       **executor)

aclnnStatus aclnnGroupedMatmulSwigluQuantV2(
    void          *workspace,
    uint64_t       workspaceSize,
    aclOpExecutor *executor,
    aclrtStream    stream)

aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
x	Input	Represents the left matrix, corresponding to X in the formula.	-	FLOAT8_E4M3FN, FLOAT8_E5M2, FLOAT4_E2M1, INT8, HIFLOAT8	ND	2	√
weight	Input	Represents the weight matrix, corresponding to W in the formula.	-	FLOAT8_E4M3FN, FLOAT8_E5M2, FLOAT4_E2M1, INT8, INT4, INT32, HIFLOAT8	ND, FRACTAL_NZ	2, 3, 4, 5	√
weightScale	Input	Represents the quantization scale factor of the right matrix, i.e., wScale in the formula.	The first axis length must be equal to the first axis dimension of weight, and the last axis length must be the same as the last axis of weight restored to ND format.	FLOAT8_E8M0, UINT64, FLOAT, FLOAT16, BFLOAT16	ND	1, 2, 3, 4	√
weightAssistMatrix	Optional input.	Represents the auxiliary matrix used in matrix multiplication, which is weightAssistMatrix in the formula.	Effective only in the A8W4 scenario. A null pointer must be passed in other scenarios. The first axis length must be equal to the first axis dimension of weight, and the last axis length must be the same as the last axis of weight restored to ND format.	FLOAT	ND	1, 2	√
bias	Optional input.	Represents the bias value for matrix multiplication computation.	Reserved input, not supported yet. Pass a null pointer.	-	-	-	-
xScale	Input	Quantization factor of the left matrix, i.e., xScale in the formula.	-	FLOAT8_E8M0, FLOAT	ND	1, 3	√
smoothScale	Optional input	Represents the smooth scale factor.	Optional in the A4W4 scenario. Pass a null pointer in other scenarios. In the A4W4 scenario, the first axis length must be equal to the first axis dimension of weight. In the A4W4 scenario, a null pointer or two shapes are supported: (E, N / 2) or (E,).	FLOAT	ND	1, 2	√
groupList	Input	Indicates the number of tokens participating in the computation for each group, which is grouplist in the formula.	The length must be equal to the first axis dimension of weight. The last value in grouplist constrains the valid portion of the output data. For details, see the computation process section in Function.	INT64	ND	1	√
dequantMode	Input	Indicates the dequantization computation type, used to determine the dequantization method for the activation matrix and weight matrix.	0 indicates per-token for the activation matrix and per-channel for the weight matrix. 1 indicates per-token for the activation matrix and per-group for the weight matrix. 2 indicates MX quantization.	INT64	-	-	-
dequantDtype	Input	Indicates the data type of the intermediate GroupedMatmul result.	0 indicates FLOAT. 1 indicates FLOAT16. 27 indicates BFLOAT16. 28 indicates UNDEFINED.	INT64	-	-	-
quantMode	Input	Indicates the quantization computation type, which determines the quantization mode of the Swiglu result.	0 indicates per-token. 1 indicates per-group. 2 indicates MX quantization.	INT64	-	-	-
groupListType	Input	Indicates the interpretation mode of the grouping, used to determine the semantics of groupList.	0 indicates cumsum mode, where each element in groupList represents the cumulative length of the current group. 1 indicates count mode, where each element in groupList represents the number of elements contained in the group.	INT64	-	-	-
tuningConfig	Optional input	Estimates the M/E size for the operator and selects different operator templates to meet performance requirements in different scenarios.	An array where the first number indicates the expected number of tokens processed by each expert, used to optimize tiling. Enabled when the right matrix uses A4W4 NZ input. Pass a null pointer for other inputs.	INT64	-	-	-
output	Output	Represents the quantized output result, i.e., Q in the formula.	-	FLOAT8_E4M3FN, FLOAT8_E5M2, FLOAT4_E2M1, INT8, HIFLOAT8	ND	2	√
outputScale	Output	Represents the quantization factor of the output, i.e., QScale in the formula.	-	FLOAT8_E8M0, FLOAT	ND	1, 3	√
workspaceSize	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
executor	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.
- weight supports NZ input transpose in A4W4 scenarios, and only non-transposed input in other scenarios. INT32 is used for adaptation purposes in A8W4 and A4W4 scenarios, where one INT32 value is interpreted as eight INT4 data. The ND data format is not supported in A8W8 scenarios.
- The dequantMode parameter is supported: values 0 and 1 are supported in A8W4 and A4W4 scenarios, and only value 0 is supported in A8W8 scenarios.
- The dequantDtype and quantMode parameters are not supported.
- x and weight do not support empty tensors.
- When weight uses NZ transpose input, only single-tensor mode is supported.
- weight, weightScale, and weightAssistMatrix support single-tensor scenarios (tensorlist length is 1) and multi-tensor scenarios (tensorlist length is greater than 1).
Ascend 950PR/Ascend 950DT:
- weight supports transposition, and only the ND format is supported.
- The dequantMode parameter is supported: in the MX quantization scenario, the value 2 is supported; in the Pertoken scenario, the value 0 is supported.
- The dequantDtype parameter is supported: in the MX quantization scenario, the value 0 is supported; in the Pertoken scenario, the values 0, 1, and 27 are supported.
- Supports the quantMode parameter: In the MX quantization scenario, the value 2 is supported; in the Pertoken scenario, the value 0 is supported.
- Only the same value for dequantMode and quantMode is supported.
- x and xScale support empty tensors with M equal to 0.
- weight and weightScale support empty tensors with N equal to 0.
- weight and weightScale currently support only a tensorlist length of 1.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Value	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The parameters x, weight, weightScale, xScale, groupList, output, and outputScale are null pointers.
ACLNN_ERR_PARAM_INVALID	161002	The data dimensions of the input x, weight, weightScale, xScale, groupList, output, and outputScale do not meet the constraints.
		The shapes of the input x, weight, weightScale, xScale, groupList, output, and outputScale data do not meet the constraint conditions.
		The format of the input x, weight, weightScale, xScale, groupList, output, or outputScale does not meet the constraints.
		The length of the input weight or weightScale tensor list is not 1.
		The input x or xScale is an empty tensor, or the input weight or weightScale is an empty tensorList.
		The number of elements in groupList is greater than the first axis length of weight.
		The input dequantMode, quantMode, or dequantDtype does not meet the constraints.
		The input bias, weightAssistMatrix, smoothScale, or tuningConfig does not meet the constraints.

aclnnGroupedMatmulSwigluQuantV2

Parameters

Parameter	Input/Output	Description
workspace	Input.	Address of the workspace memory applied on the Device side.
workspaceSize	Input.	Size of the workspace applied on the Device side, obtained through the first-phase API aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize.
executor	Input	Operator executor, which contains the operator computation flow.
stream	Input	Specifies the stream for executing the task.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnGroupedMatmulSwigluQuantV2 defaults to a deterministic implementation.

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.

In A8W8/A8W4/A4W4 quantization scenarios, the following constraints must be met:

The data types must meet the following table:

Quantization Scenario	x	weight	weightScale	weightAssistMatrix	xScale	smoothScale	output	outputScale
A8W8	INT8	INT8	FLOAT, FLOAT16, BFLOAT16	nullptr	FLOAT	nullptr	INT8	FLOAT
A8W4	INT8	INT4, INT32	UINT64	FLOAT	FLOAT	nullptr	INT8	FLOAT
A4W4	INT4, INT32	INT4, INT32	UINT64	nullptr	FLOAT	nullptr/FLOAT	INT8	FLOAT

Shape constraints must meet the following table:

Quantization Scenario	x	weight	Weight scale factor	Weight assist matrix	x scale factor	Smooth scale factor	Output	outputScale
A8W8	(M, K)	NZ format shape in the form of {(E, N / 32, K / 16, 16, 32)}	{(E, N)}	nullptr	(M,)	nullptr	(M, N / 2)	(M,)
A8W4	(M, K)	In ND format, the shape is in the form of {(E, K, N)}. In NZ format with INT4, the shape is in the form of {(E, N / 64, K / 16, 16, 64)}. In NZ format with INT32, the shape is in the form of {(E, N / 64, K / 16, 16, 8)}.	In the per-channel scenario, the shape is in the form of {(E, N)}. In the per-group scenario, the shape is in the form of {(E, K_group_num, N)}.	{(E, N)}	(M,)	nullptr	(M, N / 2)	(M,)
A4W4	(M, K)	ND format shape is in the form {(E, K, N)} A4W4 supports non-transposed and transposed NZ NZ non-transposed format with INT4 has shape in the form {(E, N / 64, K / 16, 16, 64)} NZ non-transposed format with INT32 has shape in the form {(E, N / 64, K / 16, 16, 8)} NZ transposed format with INT4 has original shape in the form {(E, K / 64, N / 16, 16, 64)}, and is passed in after calling transpose(-1,-2) NZ transposed format with INT32 has original shape in the form {(E, K / 64, N / 16, 16, 8)}, and is passed in after calling transpose(-1,-2) For NZ transposed input, ensure that per-group K/K_group_num is aligned to 64	Per-channel scenario shape is in the form {(E, N)} Per-group scenario shape is in the form {(E, K_group_num, N)}	nullptr	(M,)	nullptr (E,) (E, N / 2)	(M, N / 2)	(M,)

In the A8W8 scenario, the N axis length cannot exceed 10240, and the last axis length of x cannot be greater than or equal to 65536.
In the A8W4 scenario, the N axis length cannot exceed 10240, and the last axis length of x cannot be greater than or equal to 20000.
In the A4W4 scenario, an N-axis length exceeding 10240 is not supported, and a last axis length of x greater than or equal to 20000 is not supported.
In the multi-tensor scenario, that is, when the tensorlist length is greater than 1, the shapes of weight, weightScale, and weightAssistMatrix need to be flattened along the E dimension. For example, {(E, K, N)} needs to become {E instances of (K, N)}.

Ascend 950PR/Ascend 950DT:

The first dimension of groupList supports a maximum of 1024, that is, a maximum of 1024 groups are supported.

The following constraints must be met in the MX quantization scenario:

The data types must meet the following table:

MX Quantization Scenario	x	weight	weightScale	xScale	output	outputScale
MXFP8	FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT8_E8M0	FLOAT8_E8M0	FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT8_E8M0
MXFP4	FLOAT4_E2M1	FLOAT4_E2M1	FLOAT8_E8M0	FLOAT8_E8M0	FLOAT4_E2M1, FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT8_E8M0

The shape constraints must meet the following table:

  <table style="undefined;table-layout: fixed; width: 1134px"><colgroup>
  <col style="width: 130px">
  <col style="width: 250px">
  <col style="width: 320px">
  <col style="width: 180px">
  <col style="width: 250px">
  <col style="width: 160px">
  </colgroup>
  <thead>
    <tr>
      <th>x</th>
      <th>weight</th>
      <th>weightScale</th>
      <th>xScale</th>
      <th>output</th>
      <th>outputScale</th>
    </tr></thead>
  <tbody>
    <tr>
      <td>(M, K)</td>
      <td><ul>
      <li>Non-transposed shape: {(E, K, N)}</li>
      <li>Transposed shape: {(E, N, K)}</li></ul></td>
      <td><ul>
      <li>Non-transposed shape: {(E, ceil(K / 64), N, 2)}</li>
      <li>Transposed shape: {(E, N, ceil(K / 64), 2)}</li></ul></td>
      <td>(M, ceil(K / 64), 2)</td>
      <td>(M, N / 2)</td>
      <td>(M, ceil((N / 2) / 64), 2)</td>
    </tr>
  </tbody>
  </table>

- The transpose attribute of weightScale must be consistent with that of weight.

- In the MX quantization scenario, N must be 128-aligned.

- The MXFP4 scenario does not support K=2.

- In the MXFP4 scenario, K must be an even number. When the data type of output is FLOAT4_E2M1, N must be an even number greater than or equal to 4.

The Pertoken quantization scenario must satisfy the following constraints:

The data types must meet the following table:

x	weight	weightScale	xScale	output	outputScale
FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT, BFLOAT16	FLOAT	FLOAT8_E4M3FN, FLOAT8_E5M2	FLOAT
INT8	INT8	FLOAT, BFLOAT16, FLOAT16	FLOAT	INT8	FLOAT
HIFLOAT8	HIFLOAT8	FLOAT, BFLOAT16	FLOAT	HIFLOAT8	FLOAT

The shape constraints must satisfy the following table:

x	weight	weightScale	xScale	output	outputScale
(M, K)	Non-transposed shape: {(E, K, N)} Transposed shape: {(E, N, K)}	Shape: {(E, N)}	(M, )	(M, N / 2)	(M, )

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul_swiglu_quant_v2.h"

#define CHECK_RET(cond, return_expr)                                                                                   \
    do {                                                                                                               \
        if (!(cond)) {                                                                                                 \
            return_expr;                                                                                               \
        }                                                                                                              \
    } while (0)

#define LOG_PRINT(message, ...)                                                                                        \
    do {                                                                                                               \
        printf(message, ##__VA_ARGS__);                                                                                \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate. Initialize resources.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape,
                    void** deviceAddr, aclDataType dataType, aclFormat formatType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy data from the host side to the device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, formatType,
                            shape.data(), shape.size(), *deviceAddr);
    return 0;
}

template <typename T>
int CreateAclTensorList(const std::vector<T> &hostData, const std::vector<std::vector<int64_t>> &shapes,
                        void **deviceAddr, aclDataType dataType, aclFormat formatType, aclTensorList **tensor) {
    int size = shapes.size();
    aclTensor* tensors[size];
    for (int i = 0; i < size; i++) {
        int ret = CreateAclTensor<T>(hostData, shapes[i], deviceAddr + i, dataType, formatType, tensors + i);
        CHECK_RET(ret == ACL_SUCCESS, return ret);
    }
    *tensor = aclCreateTensorList(tensors, size);
    return ACL_SUCCESS;
}

int main() {
    // 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
    // Set deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check as needed.
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Customize the construction based on the API interface.
    int64_t E = 4;
    int64_t M = 8192;
    int64_t N = 4096;
    int64_t K = 7168;
    std::vector<int64_t> xShape = {M, K};
    std::vector<std::vector<int64_t>> weightShape = {{E, N / 32 , K / 16, 16, 32}};
    std::vector<std::vector<int64_t>> weightScaleShape = {{E, N}};
    std::vector<int64_t> xScaleShape = {M};
    std::vector<int64_t> groupListShape = {E};
    std::vector<int64_t> outputShape = {M, N / 2};
    std::vector<int64_t> outputScaleShape = {M};

    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr[1];
    void* weightScaleDeviceAddr[1];
    void* xScaleDeviceAddr = nullptr;
    void* groupListDeviceAddr = nullptr;
    void* outputDeviceAddr = nullptr;
    void* outputScaleDeviceAddr = nullptr;

    aclTensor* x = nullptr;
    aclTensorList* weight = nullptr;
    aclTensorList* weightScale = nullptr;
    aclTensor* xScale = nullptr;
    aclTensor* groupList = nullptr;
    aclTensor* output = nullptr;
    aclTensor* outputScale = nullptr;

    std::vector<int8_t> xHostData(M * K, 0);
    std::vector<int8_t> weightHostData(E * N * K, 0);
    std::vector<float> weightScaleHostData(E * N, 0);
    std::vector<float> xScaleHostData(M, 0);
    std::vector<int64_t> groupListHostData(E, 0);
    std::vector<int8_t> outputHostData(M * N / 2, 0);
    std::vector<float> outputScaleHostData(M, 0);

    // Create the x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_ND, &x);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the weight aclTensorList.
    ret = CreateAclTensorList(weightHostData, weightShape, weightDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_FRACTAL_NZ, &weight);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the weightScale aclTensorList.
    ret = CreateAclTensorList(weightScaleHostData, weightScaleShape, weightScaleDeviceAddr, aclDataType::ACL_FLOAT,  aclFormat::ACL_FORMAT_ND, &weightScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the xScale aclTensor.
    ret = CreateAclTensor(xScaleHostData, xScaleShape, &xScaleDeviceAddr, aclDataType::ACL_FLOAT, aclFormat::ACL_FORMAT_ND, &xScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the groupList aclTensor.
    ret = CreateAclTensor(groupListHostData, groupListShape, &groupListDeviceAddr, aclDataType::ACL_INT64, aclFormat::ACL_FORMAT_ND, &groupList);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the output aclTensor.
    ret = CreateAclTensor(outputHostData, outputShape, &outputDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_ND, &output);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the outputScale aclTensor.
    ret = CreateAclTensor(outputScaleHostData, outputScaleShape, &outputScaleDeviceAddr, aclDataType::ACL_FLOAT, aclFormat::ACL_FORMAT_ND, &outputScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // New V2 parameter.
    aclTensorList* weightAssistMatrix = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* smoothScale = nullptr;
    int64_t dequantMode = 0;
    int64_t dequantDtype = 28;
    int64_t quantMode = 0;
    int64_t groupListType = 0;

    std::vector<int64_t> tuningConfigData;
    aclIntArray* tuningConfig = aclCreateIntArray(tuningConfigData.data(), 0);

    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;

    // 3. Call the CANN operator library API.
    // Call the first-phase API of aclnnGroupedMatmulSwigluQuantV2.
    ret = aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize(
        x, weight, weightScale, weightAssistMatrix, bias, xScale, smoothScale, groupList, dequantMode, dequantDtype,
        quantMode, groupListType, tuningConfig, output, outputScale, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS,
    LOG_PRINT("aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnGroupedMatmulSwigluQuantV2.
    ret = aclnnGroupedMatmulSwigluQuantV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS,
    LOG_PRINT("aclnnGroupedMatmulSwigluQuantV2 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the Device memory to the Host memory. Modify based on the specific API definition.
    auto size = 10;
    std::vector<int8_t> out1Data(size, 0);
    ret = aclrtMemcpy(out1Data.data(), out1Data.size() * sizeof(out1Data[0]), outputDeviceAddr,
                        size * sizeof(out1Data[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %d\n", j, out1Data[j]);
    }
    std::vector<float> out2Data(size, 0);
    ret = aclrtMemcpy(out2Data.data(), out2Data.size() * sizeof(out2Data[0]), outputScaleDeviceAddr,
                        size * sizeof(out2Data[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %f\n", j, out2Data[j]);
    }
    // 6. Release aclTensor, aclTensorList, and aclScalar. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyTensorList(weight);
    aclDestroyTensorList(weightScale);
    aclDestroyTensor(xScale);
    aclDestroyTensor(groupList);
    aclDestroyTensor(output);
    aclDestroyTensor(outputScale);

    aclDestroyIntArray(tuningConfig);

    // 7. Release device resources. Modify based on the specific API definition.
    aclrtFree(xDeviceAddr);
    for (int64_t i = 0; i < 1; i++) {
        aclrtFree(weightDeviceAddr[i]);
        aclrtFree(weightScaleDeviceAddr[i]);
    }
    aclrtFree(xScaleDeviceAddr);
    aclrtFree(groupListDeviceAddr);
    aclrtFree(outputDeviceAddr);
    aclrtFree(outputScaleDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}

Ascend 950PR/Ascend 950DT:

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul_swiglu_quant_v2.h"

#define CHECK_RET(cond, return_expr) \
    do {                               \
      if (!(cond)) {                   \
        return_expr;                   \
      }                                \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)     \
    do {                              \
      printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate. Initialize resources.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
                        aclDataType dataType, aclFormat FormatType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);

    // Call aclrtMemcpy to copy host data to device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, FormatType,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

template <typename T>
int CreateAclTensorList(const std::vector<std::vector<T>>& hostData, const std::vector<std::vector<int64_t>>& shapes,
                        void** deviceAddr, aclDataType dataType, aclTensorList** tensor) {
    int size = shapes.size();
    aclTensor* tensors[size];
    for (int i = 0; i < size; i++) {
        int ret = CreateAclTensor<T>(hostData[i], shapes[i], deviceAddr + i, dataType, ACL_FORMAT_ND, tensors + i);
        CHECK_RET(ret == ACL_SUCCESS, return ret);
    }
    *tensor = aclCreateTensorList(tensors, size);
    return ACL_SUCCESS;
}

template <typename T1, typename T2>
auto CeilDiv(T1 a, T2 b) -> T1
{
    if (b == 0) {
        return a;
    }
    return (a + b - 1) / b;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int aclnnGroupedMatmulSwigluQuantV2Test(int32_t deviceId, aclrtStream& stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Custom construction is required based on the API.
    int64_t E = 8;
    int64_t M = 2048;
    int64_t N = 4096;
    int64_t K = 7168;

    std::vector<int64_t> xShape = {M, K};
    std::vector<int64_t> weightShape = {E, K, N};
    std::vector<int64_t> weightScaleShape = {E, CeilDiv(K, 64), N, 2};
    std::vector<int64_t> xScaleShape = {M, CeilDiv(K, 64), 2};
    std::vector<int64_t> groupListShape = {E};
    std::vector<int64_t> outputShape = {M, N / 2};
    std::vector<int64_t> outputScaleShape = {M, CeilDiv((N / 2), 64), 2};

    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* weightScaleDeviceAddr = nullptr;
    void* xScaleDeviceAddr = nullptr;
    void* groupListDeviceAddr = nullptr;
    void* outputDeviceAddr = nullptr;
    void* outputScaleDeviceAddr = nullptr;

    aclTensor* x = nullptr;
    aclTensorList* weight = nullptr;
    aclTensorList* weightScale = nullptr;
    aclTensor* xScale = nullptr;
    aclTensor* groupList = nullptr;
    aclTensor* output = nullptr;
    aclTensor* outputScale = nullptr;
    aclTensorList* weightAssistMatri = nullptr;
    aclTensorList* smoothScale = nullptr;

    std::vector<int8_t> xHostData(M * K, 1);
    std::vector<int8_t> weightHostData(E * N * K, 1);
    std::vector<int8_t> weightScaleHostData(E * CeilDiv(K, 64) * N * 2, 1);
    std::vector<int8_t> xScaleHostData(M * CeilDiv(K, 64) * 2, 1);
    std::vector<int64_t> groupListHostData(E, 1);
    std::vector<int8_t> outputHostData(M * N / 2, 1);
    std::vector<int8_t> outputScaleHostData(M * CeilDiv((N / 2), 64) * 2, 1);
    std::vector<int64_t> tuningConfigData = {1};
    aclIntArray *tuningConfig = aclCreateIntArray(tuningConfigData.data(), 1);

    int64_t quantMode = 2;
    int64_t dequantMode = 2;
    int64_t dequantDtype = 0;
    int64_t groupListType = 1;

    // Create the x aclTensor.
    std::vector<unsigned char> xHostDataUnsigned(xHostData.begin(), xHostData.end());
    ret = CreateAclTensor<uint8_t>(xHostDataUnsigned, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT8_E5M2, aclFormat::ACL_FORMAT_ND, &x);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> xTensorPtr(x, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> xDeviceAddrPtr(xDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the weight aclTensorList.
    std::vector<std::vector<int8_t>> weightHostDataList = {weightHostData};
    std::vector<std::vector<int64_t>> weightShapeList = {weightShape};
    ret = CreateAclTensorList<int8_t>(weightHostDataList, weightShapeList, &weightDeviceAddr, aclDataType::ACL_FLOAT8_E5M2, &weight);
    std::unique_ptr<aclTensorList, aclnnStatus (*)(const aclTensorList*)> weightTensorListPtr(weight, aclDestroyTensorList);
    std::unique_ptr<void, aclError (*)(void*)> weightDeviceAddrPtr(weightDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the weightScale aclTensorList.
    std::vector<std::vector<int8_t>> weightScaleHostDataList = {weightScaleHostData};
    std::vector<std::vector<int64_t>> weightScaleShapeList = {weightScaleShape};
    ret = CreateAclTensorList<int8_t>(weightScaleHostDataList, weightScaleShapeList, &weightScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, &weightScale);
    std::unique_ptr<aclTensorList, aclnnStatus (*)(const aclTensorList*)> weightScaleTensorListPtr(weightScale, aclDestroyTensorList);
    std::unique_ptr<void, aclError (*)(void*)> weightScaleDeviceAddrPtr(weightScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the xScale aclTensor.
    ret = CreateAclTensor<int8_t>(xScaleHostData, xScaleShape, &xScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, aclFormat::ACL_FORMAT_ND, &xScale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> xScaleTensorPtr(xScale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> xScaleDeviceAddrPtr(xScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the group_list aclTensor.
    ret = CreateAclTensor<int64_t>(groupListHostData, groupListShape, &groupListDeviceAddr, aclDataType::ACL_INT64, aclFormat::ACL_FORMAT_ND, &groupList);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> groupListTensorPtr(groupList, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> groupListDeviceAddrPtr(groupListDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the y aclTensor.
    ret = CreateAclTensor<int8_t>(outputHostData, outputShape, &outputDeviceAddr, aclDataType::ACL_FLOAT8_E5M2, aclFormat::ACL_FORMAT_ND, &output);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outputTensorPtr(output, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outputDeviceAddrPtr(outputDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the yScale aclTensor.
    ret = CreateAclTensor<int8_t>(outputScaleHostData, outputScaleShape, &outputScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, aclFormat::ACL_FORMAT_ND, &outputScale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outputScaleTensorPtr(outputScale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outputScaleDeviceAddrPtr(outputScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    void* workspaceAddr = nullptr;

    // 3. Call the CANN operator library API.
    // Call the first-phase API of aclnnGroupedMatmulSwigluQuantV2.
    ret = aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize(x, weight, weightScale, nullptr, nullptr, xScale, nullptr, groupList,
                                                        dequantMode, dequantDtype, quantMode, groupListType, nullptr, output, outputScale, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulSwigluQuantV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize computed by the first-phase API.
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnGroupedMatmulSwigluQuantV2.
    ret = aclnnGroupedMatmulSwigluQuantV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulSwigluQuantV2 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the Device memory to the Host. Modify this step based on the specific API definition.
    auto size = GetShapeSize(outputShape);
    std::vector<int8_t> outputData(size, 0);
    ret = aclrtMemcpy(outputData.data(), size * sizeof(outputData[0]), outputDeviceAddr,
                      size * sizeof(outputData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy outputData from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %d\n", j, outputData[j]);
    }

    size = GetShapeSize(outputScaleShape);
    std::vector<int8_t> outputScaleData(size, 0);
    ret = aclrtMemcpy(outputScaleData.data(), size * sizeof(outputScaleData[0]), outputScaleDeviceAddr,
                      size * sizeof(outputScaleData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy outputScaleData from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %d\n", j, outputScaleData[j]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // (Boilerplate) Initialize the device/stream. See the AscendCL external API list.
    // Set deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnGroupedMatmulSwigluQuantV2Test(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulSwigluQuantV2Test failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}