aclnnGroupedMatmulSwigluQuantWeightNzV2

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	×
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Fuses GroupedMatmul, dequant, swiglu, and quant. For details, see the formula. This is a weightNZ-specific version of the aclnnGroupedMatmulSwigluQuantV2 API. The difference between this API and aclnnGroupedMatmulSwigluQuantV2 is that the weight parameter is forcibly treated as FRACTAL_NZ format in this scenario.
Formula:
- x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.
  Quantization scenario A8W8 (A refers to the activation matrix, W refers to the weight matrix, and 8 refers to the INT8 data type):
  - Definition:
    - ⋅ denotes matrix multiplication.
    - ⊙ denotes per-element multiplication.
    - $\left \lfloor x\right \rceil$ denotes rounding x to the nearest integer.
    - $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
    - $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
  - Input:
    - $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
    - $W∈\mathbb{Z_8}^{E \times K \times N}$ : grouped weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
    - $w\_scale∈\mathbb{R}^{E \times N}$ : per-channel scale factor of the grouped weight matrix (right matrix), where E is the number of experts and N is the output dimension.
    - $x\_scale∈\mathbb{R}^{M}$ : Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.
    - $grouplist∈\mathbb{N}^{E}$ : Group index list of cumsum or count.
  - Output:
    - $Q∈\mathbb{Z_8}^{M \times N / 2}$ : Quantized output matrix.
    - $Q\_scale∈\mathbb{R}^{M}$ : Quantization scale factor.
  - Computation Process
    - Determine the tokens of the current group based on groupList[i], where $i \in [0, Len(groupList)]$ .
      
      Example: Assume groupList = [3, 4, 4, 6] with groupListType = cumsum, or groupList = [3, 1, 0, 2] with groupListType = count.
      
      Note: The two different grouping methods above actually produce the same grouping result.
      
      The 0th right matrix W[0,:,:] corresponds to tokens x[0:3] at index positions [0, 3) (3 - 0 = 3 tokens in total), and corresponds to x_scale[0:3], w_scale[0], bias[0], offset[0], Q[0:3], Q_scale[0:3], and Q_offset[0:3].
      
      The 1st right matrix W[1,:,:] corresponds to tokens x[3:4] at index positions [3, 4) (4 - 3 = 1 token in total), and corresponds to x_scale[3:4], w_scale[1], bias[1], offset[1], Q[3:4], Q_scale[3:4], and Q_offset[3:4].
      
      The 2nd right matrix W[2,:,:] corresponds to tokens x[4:4] at index positions [4, 4) (4 - 4 = 0 tokens in total), and corresponds to x_scale[4:4], w_scale[2], bias[2], offset[2], Q[4:4], Q_scale[4:4], and Q_offset[4:4].
      
      The 3rd right matrix W[3,:,:] corresponds to tokens x[4:6] at index positions [4, 6) (6 - 4 = 2 tokens in total), and corresponds to x_scale[4:6], w_scale[3], bias[3], offset[3], Q[4:6], Q_scale[4:6], and Q_offset[4:6].
      
      Note: Portions not specified in groupList will not participate in the update. For example, when groupList = [12, 14, 18] and GroupListType = cumsum, and the shape of X is [30, :]:
      
      The shape of the first output Q is [30, :], where the portion Q[18:, :] will not be updated or initialized, and the data in that portion is the original data from when the memory space was allocated.
      
      Similarly, the shape of the second output Q is [30], where the portion Q_scale[18:] will not be updated or initialized, and the data in that portion is the original data from when the memory space was allocated.
      
      That is, Q[:grouplist[-1], :] and Q_scale[:grouplist[-1]] in the output are the valid data portions.
    - Perform the following computation based on the input parameters determined by the grouping:
      
      $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$
      
      $C_{i,act}, gate_{i} = split(C_{i})$
      
      $S_{i}=Swish(C_{i,act})\odot gate_{i}$ where $Swish(x)=\frac{x}{1+e^{-x}}$
    - Quantized output result
      
      $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$
      
      $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

MSD scenario A8W4 (A refers to the activation matrix, W refers to the weight matrix, 8 refers to the INT8 data type, and 4 refers to the INT4 data type):

Definition:
- ⋅ denotes matrix multiplication.
- ⊙ denotes per-element multiplication.
- $\left \lfloor x\right \rceil$ denotes rounding x to the nearest integer.
- $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
- $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
- $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
Input:
- $X∈\mathbb{Z_8}^{M \times K}$ : activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
- $W∈\mathbb{Z_4}^{E \times K \times N}$ : grouped weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
- $weightAssistMatrix∈\mathbb{R}^{E \times N}$ : Auxiliary matrix used in matrix multiplication (see the computation process for generating the auxiliary matrix below).
- $w\_scale∈\mathbb{R}^{E \times K\_group\_num \times N}$ : Per-channel scale factor of the grouped weight matrix (right matrix), where E is the number of experts, K_group_num is the number of groups along the K axis, and N is the output dimension.
- $x\_scale∈\mathbb{R}^{M}$ : Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.
- $grouplist∈\mathbb{N}^{E}$ : Group index list of cumsum or count.
Output:
- $Q∈\mathbb{Z_8}^{M \times N / 2}$ : Quantized output matrix.
- $Q\_scale∈\mathbb{R}^{M}$ : Quantization scale factor.
Computation Process
- 1. Determine the tokens of the current group based on groupList[i], where $i \in [0,Len(groupList)]$ .
  - The grouping logic is the same as that of A8W8.
- 1. Computation process for generating the auxiliary matrix (weightAssistMatrix) (note that the weightAssistMatrix is partially computed offline and provided as input, not completed inside the operator):
  - For per-channel quantization ( $w\_scale$ is 2-dimensional):
    
    $weightAssistMatrix_{i} = 8 × weightScale × Σ_{k=0}^{K-1} weight[:,k,:]$
  - For per-group quantization ( $w\_scale$ is 3-dimensional):
    
    $weightAssistMatrix_{i} = 8 × Σ_{k=0}^{K-1} (weight[:,k,:] × weightScale[:, ⌊k/num\_per\_group⌋, :])$
    
    NOTE: $num\_per\_group = K // K\_group\_num$
- 1. Perform the following computation based on the inputs determined by grouping:
  - 3.1. Convert the left matrix $\mathbb{Z_8}$ into high and low 4-bit parts $\mathbb{Z_4}$ :
    
    $X\_high\_4bits_{i} = \lfloor \frac{X_{i}}{16} \rfloor$ $X\_low\_4bits_{i} = X_{i} \& 0x0f - 8$
  - 3.2. Enable per-channel or per-group quantization during matrix multiplication.
    
    per-channel:
    
    $C\_high_{i} = (X\_high\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$
    
    $C\_low_{i} = (X\_low\_4bits_{i} \cdot W_{i}) \odot w\_scale_{i}$
    
    per-group:
    
    $C\_high_{i} = \\ Σ_{k=0}^{K-1}((X\_high\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$
    
    $C\_low_{i} = \\ Σ_{k=0}^{K-1}((X\_low\_4bits_{i}[:, k * num\_per\_group : (k+1) * num\_per\_group] \cdot W_{i}[k * num\_per\_group : (k+1) * num\_per\_group, :]) \odot w\_scale_{i}[k, :] )$
  - 3.3. Restore the high-bit and low-bit matrix multiplication results to the overall result.
    
    $C_{i} = (C\_high_{i} * 16 + C\_low_{i} + weightAssistMatrix_{i}) \odot x\_scale_{i}$
    
    $C_{i,act}, gate_{i} = split(C_{i})$
    
    $S_{i}=Swish(C_{i,act})\odot gate_{i}$ where $Swish(x)=\frac{x}{1+e^{-x}}$
- 1. Quantize the output result
  $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$
  
  $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

Quantization scenario A4W4 (A refers to the activation matrix, W refers to the weight matrix, and 4 refers to the INT4 data type):

Definition:
- ⋅ denotes matrix multiplication.
- ⊙ indicates per-element multiplication.
- $\left \lfloor x\right \rceil$ indicates rounding x to the nearest integer.
- $\mathbb{Z_4} = \{ x \in \mathbb{Z} | −8≤x≤7 \}$
- $\mathbb{Z_8} = \{ x \in \mathbb{Z} | −128≤x≤127 \}$
- $\mathbb{Z_{32}} = \{ x \in \mathbb{Z} | -2147483648≤x≤2147483647 \}$
Input:
- $X∈\mathbb{Z_4}^{M \times K}$ : Activation matrix (left matrix), where M is the total number of tokens and K is the feature dimension.
- $W∈\mathbb{Z_4}^{E \times K \times N}$ : Grouped weight matrix (right matrix), where E is the number of experts, K is the feature dimension, and N is the output dimension.
- $w\_scale∈\mathbb{R}^{E \times N}$ : Per-channel scale factor of the grouped weight matrix (right matrix), where E is the number of experts and N is the output dimension.
- $x\_scale∈\mathbb{R}^{M}$ : Per-token scale factor of the activation matrix (left matrix), where M is the total number of tokens.
- $smoothScale∈\mathbb{R}^{E \times N/2}(per-channel) or \mathbb{R}^{E}(per-tensor)$ : smooth scale factor, where E is the number of experts and N is the output dimension.
- $grouplist∈\mathbb{N}^{E}$ : group index list of cumsum or count.
Output:
- $Q∈\mathbb{Z_8}^{M \times N / 2}$ : quantized output matrix.
- $Q\_scale∈\mathbb{R}^{M}$ : quantization scale factor.
Computation Process
- 1. Determine the tokens of the current group based on groupList[i], where $i \in [0,Len(groupList)]$ .
  - The grouping logic is the same as that of A8W8.
- 1. Perform the following computation based on the input parameters determined by the group:
  $C_{i} = (X_{i}\cdot W_{i} )\odot x\_scale_{i\ Broadcast} \odot w\_scale_{i\ Broadcast}$
  
  $C_{i,act}, gate_{i} = split(C_{i})$
  
  $S_{i}=Swish(C_{i,act})\odot gate_{i}$ where $Swish(x)=\frac{x}{1+e^{-x}}$
  
  $S_{i} = S_{i} \odot smoothScale_{i\ Broadcast}$
- 1. Quantize the output result
  $Q\_scale_{i} = \frac{max(|S_{i}|)}{127}$
  
  $Q_{i} = \left\lfloor \frac{S_{i}}{Q\_scale_{i}} \right\rceil$

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call the "aclnnGroupedMatmulSwigluQuantWeightNzV2" API to execute the computation.

aclnnStatus aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize(
    const aclTensor     *x,
    const aclTensorList *weight,
    const aclTensorList *weightScale,
    const aclTensorList *weightAssistMatrix,
    const aclTensor     *bias,
    const aclTensor     *xScale,
    const aclTensor     *smoothScale,
    const aclTensor     *groupList,
    int64_t              dequantMode,
    int64_t              dequantDtype,
    int64_t              quantMode,
    int64_t              groupListType,
    const aclIntArray   *tuningConfig,
    aclTensor           *output,
    aclTensor           *outputScale,
    uint64_t            *workspaceSize,
    aclOpExecutor       **executor)

aclnnStatus aclnnGroupedMatmulSwigluQuantWeightNzV2(
    void          *workspace,
    uint64_t       workspaceSize,
    aclOpExecutor *executor,
    aclrtStream    stream)

aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-Contiguous Tensor
x	Input	Represents the left matrix, corresponding to X in the formula.	-	INT8, INT4, INT32	ND	2	√
weight	Input	Represents the weight matrix, corresponding to W in the formula.	Forcibly treats weight as FRACTAL_NZ format in this API.	INT8, INT4, INT32	FRACTAL_NZ	4, 5	√
weightScale	Input	Represents the quantization factor of the right matrix, i.e., wScale in the formula.	The first axis length must be equal to the first axis dimension of weight, and the last axis length must be the same as the last axis of weight restored to ND format.	UINT64, FLOAT, FLOAT16, BFLOAT16	ND	1, 2, 3	√
weightAssistMatrix	Optional input	Represents the auxiliary matrix used in matrix multiplication, i.e., weightAssistMatrix in the formula.	Effective only in the A8W4 scenario; a null pointer must be passed in other scenarios. The first axis length must be equal to the first axis dimension of weight, and the last axis length must be the same as the last axis of weight restored to ND format.	FLOAT	ND	1, 2	√
bias	Optional input	Represents the bias value for matrix multiplication computation.	Reserved input, not supported yet. Pass a null pointer.	-	-	-	-
xScale	Input	Quantization scale factor of the left matrix, i.e., xScale in the formula.	-	FLOAT	ND	1	√
smoothScale	Optional input	Represents the smooth scale factor.	Optional in the A4W4 scenario; a null pointer must be passed in other scenarios. In the A4W4 scenario, the first axis length must be equal to the first axis dimension of weight. In the A4W4 scenario, a null pointer or two shapes are supported: (E, N / 2) or (E,).	FLOAT	ND	1, 2	√
groupList	Input	Indicates the number of tokens participating in the computation for each group, i.e., grouplist in the formula.	The length must be equal to the first axis dimension of weight. The last value in grouplist constrains the valid portion of the output data. For details, see the computation process in Function.	INT64	ND	1	√
dequantMode	Input	Indicates the dequantization computation type, used to determine the dequantization mode for the activation matrix and weight matrix.	0 indicates per-token for the activation matrix and per-channel for the weight matrix. 1 indicates per-token for the activation matrix and per-group for the weight matrix.	INT64	-	-	-
dequantDtype	Input	Indicates the data type of the intermediate GroupedMatmul result.	Not supported yet. Defaults to 28. 0 indicates FLOAT. 1 indicates FLOAT16. 27 indicates BFLOAT16. 28 indicates UNDEFINED.	INT64	-	-	-
quantMode	Input	Indicates the quantization computation type, used to determine the quantization mode of the swiglu result.	Not supported yet. Defaults to 0. 0 indicates per-token.	INT64	-	-	-
groupListType	Input	Indicates the interpretation method of the groups, used to determine the semantics of groupList.	0 indicates cumsum mode, where each element in groupList represents the cumulative length of the current group. 1 indicates count mode, where each element in groupList represents the number of elements contained in the group.	INT64	-	-	-
tuningConfig	Optional input	Used to estimate the M/E size of the operator and select different operator templates to adapt to performance requirements in different scenarios.	Array. The first number passed in indicates the expected number of tokens processed by each expert, used to optimize tiling. Enabled for A4W4 right matrix NZ input. For other inputs, pass a null pointer.	INT64	-	-	-
output	Output	Represents the quantized result of the output, i.e., Q in the formula.	-	INT8	ND	2	√
outputScale	Output	Represents the quantization factor of the output, i.e., QScale in the formula.	-	FLOAT	ND	1	√
workspaceSize	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
executor	Output	Returns the operator executor, which contains the operator computation process.	-	-	-	-	-

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.
- weight is forcibly treated as FRACTAL_NZ format.
- weight supports transposition in A4W4, while other inputs only support non-transposed. INT32 is used for adaptation purposes in A8W4 and A4W4 scenarios, where one INT32 is interpreted as eight INT4 data. The A8W8 scenario does not support ND data format.
- The dequantMode parameter is supported: values 0 and 1 are supported in A8W4 and A4W4 scenarios, while only value 0 is supported in the A8W8 scenario.
- The dequantDtype and quantMode parameters are not supported.
- x and weight do not support empty tensors.
- When weight is input in NZ transpose format, only single-tensor mode is supported.
- weight, weightScale, and weightAssistMatrix support single-tensor scenarios (tensorlist length is 1) and multi-tensor scenarios (tensorlist length is greater than 1).

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Value	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The parameters x, weight, weightScale, xScale, groupList, output, and outputScale are null pointers.
ACLNN_ERR_PARAM_INVALID	161002	The data dimensions of the input x, weight, weightScale, xScale, groupList, output, and outputScale do not meet the constraint conditions.
		The shapes of the input x, weight, weightScale, xScale, groupList, output, and outputScale do not meet the constraint conditions.
		The data formats of the input x, weight, weightScale, xScale, groupList, output, and outputScale do not meet the constraint conditions.
		The tensor list length of the input weight and weightScale is not 1.
		The input x and xScale are empty tensors, and the input weight and weightScale are empty tensor lists.
		The number of elements in groupList exceeds the first axis length of weight.
		The passed dequantMode, quantMode, or dequantDtype does not meet the constraint conditions.
		The passed bias, weightAssistMatrix, smoothScale, or tuningConfig does not meet the constraint conditions.

aclnnGroupedMatmulSwigluQuantWeightNzV2

Parameters

Parameter	Input/Output	Description
workspace	Input	Workspace memory address applied for on the Device side.
workspaceSize	Input	Workspace size applied for on the Device side, obtained through the first-phase API aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize.
Executor	Input	Operator executor, which contains the operator computation process.
Stream	Input	Specifies the stream for task execution.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation:
- aclnnGroupedMatmulSwigluQuantWeightNzV2 defaults to a deterministic implementation.

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.

The following constraint conditions must be met in A8W8/A8W4/A4W4 quantization scenarios:

The data types must meet the requirements in the following table:

Quantization Scenario	x	weight	weightScale	weightAssistMatrix	xScale	smoothScale	output	outputScale
A8W8	INT8	INT8	FLOAT, FLOAT16, BFLOAT16	nullptr	FLOAT	nullptr	INT8	FLOAT
A8W4	INT8	INT4, INT32	UINT64	FLOAT	FLOAT	nullptr	INT8	FLOAT
A4W4	INT4, INT32	INT4, INT32	UINT64	nullptr	FLOAT	nullptr/FLOAT	INT8	FLOAT

The shape constraints must meet the following table:

Quantization Scenario	x	weight	Weight Scale	Weight Assist Matrix	X Scale	Smooth Scale	Output	outputScale
A8W8	(M, K)	NZ format shape in the form of {(E, N / 32, K / 16, 16, 32)}	{(E, N)}	nullptr	(M,)	nullptr	(M, N / 2)	(M,)
A8W4	(M, K)	ND format: shape in the form of {(E, K, N)}. NZ format with INT4: shape in the form of {(E, N / 64, K / 16, 16, 64)}. NZ format with INT32: shape in the form of {(E, N / 64, K / 16, 16, 8)}.	Per-channel scenario: shape in the form of {(E, N)}. Per-group scenario: shape in the form of {(E, K_group_num, N)}.	{(E, N)}	(M,)	nullptr	(M, N / 2)	(M,)
A4W4	(M, K)	ND format shape is {(E, K, N)}. A4W4 supports non-transposed and transposed NZ. For NZ non-transposed format with INT4, the shape is {(E, N / 64, K / 16, 16, 64)}. For NZ non-transposed format with INT32, the shape is {(E, N / 64, K / 16, 16, 8)}. For NZ transposed format with INT4, the original shape is {(E, K / 64, N / 16, 16, 64)}, and transpose(-1, -2) is called before passing in. For NZ transposed format with INT32, the original shape is {(E, K / 64, N / 16, 16, 8)}, and transpose(-1, -2) is called before passing in. For NZ transposed input, the per-group K/K_group_num must be aligned to 64.	For per-channel scenario, the shape is {(E, N)}. For per-group scenario, the shape is {(E, K_group_num, N)}.	nullptr	(M,)	nullptr (E,) (E, N / 2)	(M, N / 2)	(M,)

In the A8W8 scenario, the N axis length cannot exceed 10240, and the last axis length of x cannot be greater than or equal to 65536.
In the A4W4 scenario, the N axis length cannot exceed 10240, and the last axis length of x cannot be greater than or equal to 20000.
In the A4W4 scenario, the N-axis length cannot exceed 10240, and the last axis length of x cannot be greater than or equal to 20000.
In the multi-tensor scenario, that is, when the tensorlist length is greater than 1, the shapes of weight, weightScale, and weightAssistMatrix need to be flattened along the E dimension. For example, {(E, K, N)} needs to become {E instances of (K, N)}.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, and x2Scale is FLOAT32.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul_swiglu_quant_weight_nz_v2.h"

#define CHECK_RET(cond, return_expr)                                                                                   \
    do {                                                                                                               \
        if (!(cond)) {                                                                                                 \
            return_expr;                                                                                               \
        }                                                                                                              \
    } while (0)

#define LOG_PRINT(message, ...)                                                                                        \
    do {                                                                                                               \
        printf(message, ##__VA_ARGS__);                                                                                \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape) {
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream) {
    // Boilerplate: Initialize resources.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape,
                    void** deviceAddr, aclDataType dataType, aclFormat formatType, aclTensor** tensor) {
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, formatType,
                            shape.data(), shape.size(), *deviceAddr);
    return 0;
}

template <typename T>
int CreateAclTensorList(const std::vector<T> &hostData, const std::vector<std::vector<int64_t>> &shapes,
                        void **deviceAddr, aclDataType dataType, aclFormat formatType, aclTensorList **tensor) {
    int size = shapes.size();
    aclTensor* tensors[size];
    for (int i = 0; i < size; i++) {
        int ret = CreateAclTensor<T>(hostData, shapes[i], deviceAddr + i, dataType, formatType, tensors + i);
        CHECK_RET(ret == ACL_SUCCESS, return ret);
    }
    *tensor = aclCreateTensorList(tensors, size);
    return ACL_SUCCESS;
}

int main() {
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Enter the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    // Handle the check based on your needs.
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Customize the construction based on the API.
    int64_t E = 4;
    int64_t M = 8192;
    int64_t N = 4096;
    int64_t K = 7168;
    std::vector<int64_t> xShape = {M, K};
    std::vector<std::vector<int64_t>> weightShape = {{E, N / 32 , K / 16, 16, 32}};
    std::vector<std::vector<int64_t>> weightScaleShape = {{E, N}};
    std::vector<int64_t> xScaleShape = {M};
    std::vector<int64_t> groupListShape = {E};
    std::vector<int64_t> outputShape = {M, N / 2};
    std::vector<int64_t> outputScaleShape = {M};

    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr[1];
    void* weightScaleDeviceAddr[1];
    void* xScaleDeviceAddr = nullptr;
    void* groupListDeviceAddr = nullptr;
    void* outputDeviceAddr = nullptr;
    void* outputScaleDeviceAddr = nullptr;

    aclTensor* x = nullptr;
    aclTensorList* weight = nullptr;
    aclTensorList* weightScale = nullptr;
    aclTensor* xScale = nullptr;
    aclTensor* groupList = nullptr;
    aclTensor* output = nullptr;
    aclTensor* outputScale = nullptr;

    std::vector<int8_t> xHostData(M * K, 1);
    std::vector<int8_t> weightHostData(E * N * K, 1);
    std::vector<float> weightScaleHostData(E * N, 0.5f);
    std::vector<float> xScaleHostData(M, 0.0314f);
    std::vector<int64_t> groupListHostData = {1, 2, 2, 3};
    std::vector<int8_t> outputHostData(M * N / 2, 0);
    std::vector<float> outputScaleHostData(M, 0);

    // Create the x aclTensor.
    ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_ND, &x);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the weight aclTensorList.
    ret = CreateAclTensorList(weightHostData, weightShape, weightDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_FRACTAL_NZ, &weight);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the weightScale aclTensorList.
    ret = CreateAclTensorList(weightScaleHostData, weightScaleShape, weightScaleDeviceAddr, aclDataType::ACL_FLOAT,  aclFormat::ACL_FORMAT_ND, &weightScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the xScale aclTensor.
    ret = CreateAclTensor(xScaleHostData, xScaleShape, &xScaleDeviceAddr, aclDataType::ACL_FLOAT, aclFormat::ACL_FORMAT_ND, &xScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the groupList aclTensor.
    ret = CreateAclTensor(groupListHostData, groupListShape, &groupListDeviceAddr, aclDataType::ACL_INT64, aclFormat::ACL_FORMAT_ND, &groupList);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the output aclTensor.
    ret = CreateAclTensor(outputHostData, outputShape, &outputDeviceAddr, aclDataType::ACL_INT8, aclFormat::ACL_FORMAT_ND, &output);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the outputScale aclTensor.
    ret = CreateAclTensor(outputScaleHostData, outputScaleShape, &outputScaleDeviceAddr, aclDataType::ACL_FLOAT, aclFormat::ACL_FORMAT_ND, &outputScale);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Add V2 parameters.
    aclTensorList* weightAssistMatrix = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* smoothScale = nullptr;
    int64_t dequantMode = 0;
    int64_t dequantDtype = 28;
    int64_t quantMode = 0;
    int64_t groupListType = 0;

    std::vector<int64_t> tuningConfigData = {};
    aclIntArray* tuningConfig = aclCreateIntArray(tuningConfigData.data(), 1);

    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;

    // 3. Call the CANN operator library API.
    // Call the first-phase API of aclnnGroupedMatmulSwigluQuantWeightNzV2.
    ret = aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize(
        x, weight, weightScale, weightAssistMatrix, bias, xScale, smoothScale, groupList, dequantMode, dequantDtype,
        quantMode, groupListType, tuningConfig, output, outputScale, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS,
    LOG_PRINT("aclnnGroupedMatmulSwigluQuantWeightNzV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnGroupedMatmulSwigluQuantWeightNzV2.
    ret = aclnnGroupedMatmulSwigluQuantWeightNzV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS,
    LOG_PRINT("aclnnGroupedMatmulSwigluQuantWeightNzV2 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
    auto size = 10;
    std::vector<int8_t> out1Data(size, 0);
    ret = aclrtMemcpy(out1Data.data(), out1Data.size() * sizeof(out1Data[0]), outputDeviceAddr,
                        size * sizeof(out1Data[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %d\n", j, out1Data[j]);
    }
    std::vector<float> out2Data(size, 0);
    ret = aclrtMemcpy(out2Data.data(), out2Data.size() * sizeof(out2Data[0]), outputScaleDeviceAddr,
                        size * sizeof(out2Data[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t j = 0; j < size; j++) {
        LOG_PRINT("result[%ld] is: %f\n", j, out2Data[j]);
    }
    // 6. Release aclTensor, aclTensorList, and aclScalar. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyTensorList(weight);
    aclDestroyTensorList(weightScale);
    aclDestroyTensor(xScale);
    aclDestroyTensor(groupList);
    aclDestroyTensor(output);
    aclDestroyTensor(outputScale);

    aclDestroyIntArray(tuningConfig);

    // 7. Release device resources. Modify based on the specific API definition.
    aclrtFree(xDeviceAddr);
    for (int64_t i = 0; i < 1; i++) {
        aclrtFree(weightDeviceAddr[i]);
        aclrtFree(weightScaleDeviceAddr[i]);
    }
    aclrtFree(weightDeviceAddr);
    aclrtFree(weightScaleDeviceAddr);
    aclrtFree(xScaleDeviceAddr);
    aclrtFree(groupListDeviceAddr);
    aclrtFree(outputDeviceAddr);
    aclrtFree(outputScaleDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}