aclnnQuantMatmulWeightNz

Applicable Products

Applicable Products	Supported
Ascend 950PR/Ascend 950DT	✓
Atlas A3 training products/Atlas A3 inference products	✓
Atlas A2 training products/Atlas A2 inference products	✓
Atlas 200I/500 A2 inference products	×
Atlas inference products	✓
Atlas training products	×

Function

Description: Performs quantized matrix multiplication. Similar APIs include aclnnMm (supports only 2D tensors as input for matrix multiplication) and aclnnBatchMatMul (supports only 3D matrix multiplication, where the first dimension is the batch dimension). Supports T-C, T-T, K-C, K-T, and mx quantization modes.
Formula:
Atlas inference products, Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT
- x1Scale without bias:
$out = x1@x2 * x2Scale + x2Offset$
- bias INT32:
$out = (x1@x2 + bias) * x2Scale + x2Offset$
- x1Scale without bias
$out = x1@x2 * x2Scale * x1Scale$
- With x1Scale, bias INT32.
$out = (x1@x2 + bias) * x2Scale * x1Scale$
Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT
- bias BFLOAT16/FLOAT32 (no x2Offset in this scenario):
$out = x1@x2 * x2Scale + bias$
- x1Scale without bias:
$out = x1@x2 * x2Scale * x1Scale$
- x1Scale, bias INT32 (no x2Offset in this scenario):
$out = (x1@x2 + bias) * x2Scale * x1Scale$
- x1Scale, bias BFLOAT16/FLOAT16/FLOAT32 (no x2Offset in this scenario):
$out = x1@x2 * x2Scale * x1Scale + bias$
- x1 is INT8, x2 is INT32, x1Scale is FLOAT32, x2Scale is UINT64, and yOffset is FLOAT32:
$out = ((x1 @ (x2*x2Scale)) + yOffset) * x1Scale$
Ascend 950PR/Ascend 950DT
- mx quantization mode: $out[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice))* (x1Scale[m/gsM, j] * x2Scale[j, n/gsN]))+bias[n]$ Where gsM, gsN, and gsK represent groupSizeM, groupSizeN, and groupSizeK respectively; x1Slice represents the vector of length groupSizeK in the m-th row of x1, and x2Slice represents the vector of length groupSizeK in the n-th column of x2. The K-axis starts slicing from j*groupSizeK, with j in the range [0, kLoops], kLoops = ceil(K / groupSizeK), where K is the length of the K-axis. The last slice may have a length less than groupSizeK.

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnQuantMatmulWeightNzGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation flow, and then call the "aclnnQuantMatmulWeightNz" API to perform the computation.

aclnnStatus aclnnQuantMatmulWeightNzGetWorkspaceSize(
    const aclTensor *x1,
    const aclTensor *x2,
    const aclTensor *x1Scale,
    const aclTensor *x2Scale,
    const aclTensor *yScale,
    const aclTensor *x1Offset,
    const aclTensor *x2Offset,
    const aclTensor *yOffset,
    const aclTensor *bias,
    bool             transposeX1,
    bool             transposeX2,
    int64_t          groupSize,
    aclTensor       *out,
    uint64_t        *workspaceSize,
    aclOpExecutor   **executor)

aclnnStatus aclnnQuantMatmulWeightNz(
    void          *workspace,
    uint64_t       workspaceSize,
    aclOpExecutor *executor,
    aclrtStream    stream)

aclnnQuantMatmulWeightNzGetWorkspaceSize

Parameters:

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (shape)	Non-contiguous Tensor
x1	Input	Input x1 in the formula.	Supports non-contiguous tensors when the last two axes are transposed. Non-contiguous tensors in other scenarios are not supported. When transposeX1 is false, each dimension represents: (batch, m, k). The batch dimension is optional. When transposeX1 is true, each dimension represents: (batch, k, m). The batch dimension is optional.	INT4^{1, 3}, INT8, FLOAT8_E4M3FN^{1, 2}	ND	2-6	✓
x2	Input	Input x2 in the formula.	When transposeX2 is true, each dimension is: (batch, k1, n1, n0, k0), where batch is optional, k0 = 32, n0 = 16. When transposeX2 is false, each dimension is: (batch, n1, k1, k0, n0), where batch is optional, k0 = 16, n0 = 32. k in x1 shape and k1 in x2 shape must satisfy ceil(k / k0) = k1, and n1 in x2 shape and n in out must satisfy ceil(n / n0) = n1.	INT4^{1, 3}, INT8, INT32^{1, 3}, FLOAT4_E2M1^{1, 2}, FLOAT32^{1, 2}, FLOAT8_E4M3FN^{1, 2}	NZ	4-8	-
x1Scale	Input	Optional quantization parameter, representing input x1Scale in the formula.	When x1Scale is FLOAT8_E8M0, x2Scale is 3-dimensional, with each dimension indicating: (m, ceil(k / 64), 2) when transposeX1 is false, and (ceil(k / 64), m, 2) when transposeX1 is true. When x1Scale is FLOAT32, its shape is 1-dimensional (t, ), where t = m, and m is consistent with the m of x1.	FLOAT32¹, FLOAT8_E8M0^{1, 2}	ND	1, 3	-
x2Scale	Input	Represents the quantization parameter, which is the input x2Scale in the formula.	When x2Scale is FLOAT8_E8M0, x2Scale is 3-dimensional, with each dimension representing: (ceil(k / 64), n, 2) when transposeX2 is false, and (n, ceil(k / 64), 2) when transposeX2 is true. When x2Scale is of another dtype, its shape is 1-dimensional (t, ), where t = 1 or n, and n is consistent with the n of x2. When the original input type does not satisfy the combinations in Constraints, call the aclnn API of the TransQuantParamV2 operator in advance to convert the scale to INT64 or UINT64 data type.	UINT64, INT64, FLOAT32¹, BFLOAT16¹, FLOAT8_E8M0^{1, 2}	ND	1, 3	-
yScale	Input	Dequantization scale parameter of the output y.	The shape is 2D (1, n), where n is the same as n of x2.	UINT64^{1, 2}, INT64^{1, 2}	ND	2	-
x1Offset	Input	Reserved parameter.	Not supported in the current version. Pass nullptr or an empty tensor.	-	-	-	-
x2Offset	Input	Optional quantization parameter, representing the input x2Offset in the formula.	shape is 1-dimensional (t, ), where t = 1 or n, and n is the same as n of x2.	FLOAT32	ND	1	-
yOffset	Input	Input yOffset in the formula.	-	FLOAT32	ND	1	-
bias	Input	Optional parameter. The input bias in the formula.	The shape supports 1D (n, ) or 3D (batch, 1, n), where n is consistent with n of x2. When the shape of out is 2D, 4D, 5D, or 6D, the shape of bias only supports 1D (n, ).	INT32, BFLOAT16¹, FLOAT16¹, FLOAT32¹	ND	1, 3	-
transposeX1	Input	Indicates whether the input shape of x1 includes transpose.	-	BOOL	-	-	-
transposeX2	Input	Indicates whether the input shape of x2 includes transpose.	-	BOOL	-	-	-
groupSize	Input	Specifies the quantization group size in the m, n, and k directions.	Concatenated from three values: groupSizeM, groupSizeN, and groupSizeK. Each value occupies 16 bits, and together they occupy the lower 48 bits of the int64_t type groupSize (the upper 16 bits are invalid). For the calculation formula, see Formula 1 below the table. Pass 0 for scenarios where groupSize is not supported.	INT64	-	-	-
out	Output	Output out in the formula.	shape supports 2 to 6 dimensions, (batch, m, n), where batch is optional. Supports broadcast of the batch dimensions of x1 and x2. The output batch is consistent with the batch after broadcast, m is consistent with m of x1, and n satisfies the relationship ceil(n / n0) = n1 with n1 and n0 of x2.	FLOAT16, INT8, BFLOAT16¹, INT32¹, FLOAT32^1、2	ND	2-6	-
workspaceSize	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
executor	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

Formula 1:
$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$
Atlas inference products
- The superscript "1" in the data type column of the table above indicates a data type not supported by this series.
- x2 does not support non-contiguous tensors.
- yScale is not supported.
- groupSize is not supported. Pass 0 for groupSize.
Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products
- The superscript "2" in the data type column of the table above indicates a data type not supported by this series.
- x2 does not support non-contiguous tensor.
- yScale is not supported.
- groupSize is not supported. Pass 0 for groupSize.
Ascend 950PR/Ascend 950DT
- The superscript "3" in the data type column of the table above indicates a data type not supported by this series.
- x2 supports non-contiguous tensors when the last two axes are transposed. Non-contiguous tensors in other scenarios are not supported.
- Passing a non-zero value for groupSize is supported.

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Value	Return Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The passed x1, x2, x2Scale, or out is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The data type or data format of x1, x2, bias, x2Scale, x2Offset, or out is not within the supported range.
		The shape of x1, x2, bias, x2Scale, x2Offset, or out does not meet the validation conditions.
		x1, x2, bias, x2Scale, x2Offset, or out is an empty tensor.
		The last dimension of x1 or x2 exceeds 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.
		The input yScale, x1Offset, or yOffset is not nullptr and is not an empty tensor.
		The passed groupSize does not meet the validation condition.

aclnnQuantMatmulWeightNz

Parameters:

Parameter	Input/Output	Description
workspace	Input	The workspace memory address applied for on the Device side.
workspaceSize	Input	The workspace size applied for on the Device side, obtained from the first-phase API aclnnQuantMatmulWeightNzGetWorkspaceSize.
executor	Input	Operator executor that contains the operator computation flow.
stream	Input	Specifies the stream that executes the task.

Return value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic description:
- aclnnQuantMatmulWeightNz defaults to a deterministic implementation.

Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products

Before calling this API, you can use aclnnTransMatmulWeight to process x2 in ND format to obtain an AI processor-friendly data layout.

The input and output support the following data type combinations:

x1	x2	x1Scale	x2Scale	x2Offset	yOffset	bias	out
INT8	INT8	null	UINT64/INT64	null	null	null/INT32	FLOAT16
INT8	INT8	null	UINT64/INT64	null/FLOAT32	null	null/INT32	INT8
INT8	INT8	null/FLOAT32	FLOAT32/BFLOAT16	null	null	null/INT32/BFLOAT16/FLOAT32	BFLOAT16
INT8	INT8	FLOAT32	FLOAT32	null	null	null/INT32/FLOAT16/FLOAT32	FLOAT16
INT8	INT8	null	FLOAT32/BFLOAT16	null	null	null/INT32	INT32
INT4	INT4	null/FLOAT32	BFLOAT16	null/FLOAT32	null	null/BFLOAT16	BFLOAT16
INT4	INT4	null/FLOAT32	FLOAT32	null/FLOAT32	null	null/BFLOAT16	BFLOAT16
INT4	INT4	null/FLOAT32	UINT64	null/FLOAT32	null	null/INT32	FLOAT16
INT4	INT4	null/FLOAT32	FLOAT32	null/FLOAT32	null	null/INT32	FLOAT16
INT8	INT32	FLOAT32	UINT64	null	FLOAT32	null	FLOAT16/BFLOAT16

Constraint on x1: When the data type is INT8 and the data type of x2 is INT32, transposeX1 must be false. Dimensions: (m, k), where k must be an even number and less than 29576.
Constraint on yOffset: The shape supports 1 dimension (n). It is an auxiliary result computed offline during the computation process, with the value required to be 8*x2*x2Scale, and accumulated along the first dimension.

Atlas inference products

Before calling this API, you can use aclnnTransMatmulWeight to process x2 with ND format to obtain an AI processor-affine data layout.
The input and output support the following data type combinations:

x1 x2 x1Scale x2Scale x2Offset bias out
INT8 INT8 null UINT64/INT64 null null/INT32 FLOAT16
INT8 INT8 null UINT64/INT64 null/FLOAT32 null/INT32 INT8
INT8 INT8 FLOAT FLOAT null null/INT32 FLOAT16
When x1Scale is not null, only K-C quantization is supported.

x1	x2	x1Scale	x2Scale	x2Offset	bias	out
INT8	INT8	null	UINT64/INT64	null	null/INT32	FLOAT16
INT8	INT8	null	UINT64/INT64	null/FLOAT32	null/INT32	INT8
INT8	INT8	FLOAT	FLOAT	null	null/INT32	FLOAT16

Ascend 950PR/Ascend 950DT

Before calling this API, you can use aclnnTransMatmulWeight or aclnnNpuFormatCast to process x2 in ND format to obtain the NZ format. During use, you must pad with 0 to prevent dirty data from being introduced.
When either of the last two dimensions of the original ND is 1, the weightNz feature cannot be used, and this API does not support such a scenario.
T-C Quantization && T-T Quantization Scenario Constraints:

- The input and output support the following data type combinations:

  | x1              | x2          | x1Scale     |   x2Scale         | x2Offset     | yScale | bias                       |   out                   |
  | --------------- | ----------- | ----------- |   --------------- | ------------ | -------| -------------------------- |   ----------------------|
  | INT8            | INT8        | null        | UINT64/INT64     | null         | null   | null/INT32                 |   FLOAT16/ BFLOAT16     |
  | INT8            | INT8        | null        | UINT64/INT64     | null/FLOAT32 | null   | null/INT32                 |   INT8                  |
  | INT8            | INT8        | null        | FLOAT32/BFLOAT16  | null         | null   | null/INT32/FLOAT32/BFLOAT16|   BFLOAT16              |
  | INT8            | INT8        | null        | FLOAT32/BFLOAT16  | null         | null   | null/INT32                 |   INT32                 |

- In the T-T quantization scenario, the shape of x1Scale is nullptr, and the shape of x2Scale is (1,).

- In the T-C quantization scenario, the shape of x1Scale is nullptr, and the shape of x2Scale is (n,), where n is the same as the n of x2.

- Dynamic T-C or dynamic T-T quantization is not supported.

K-C Quantization && K-T Quantization Scenario Constraints:

- The input and output support the following data type combinations:

  | x1                   | x2                   | x1Scale | x2Scale         | x2Offset | yScale |   bias                      | out             |
  | -------------------- | -------------------- | ------- | --------------- | -------- | -------|   ------------------------- | --------------- |
  | INT8                 | INT8                 | FLOAT32 | FLOAT32/BFLOAT16| null     | null   | null/INT32/FLOAT32/BFLOAT16 | BFLOAT16        |
  | INT8                 | INT8                 | FLOAT32 | FLOAT32         | null     | null   | null/INT32/FLOAT32/FLOAT16  | FLOAT16         |

- In the K-C quantization scenario, the shape of x1Scale is (m,), and the shape of x2Scale is (n,), where m matches the m of x1, and n matches the n of x2.

- In the K-T quantization scenario, the shape of x1Scale is (m,), and the shape of x2Scale is (1,), where m matches the m of x1.

mx quantization scenario constraints:

- The input and output support the following data type combinations:

  | x1            | x2            | x1Scale     | x2Scale     | x2Offset | yScale | bias         | out                         |
  |---------------| ------------- | ----------- | ----------- | -------- | ------ | -------------| --------------------------- |
  | FLOAT8_E4M3FN | FLOAT8_E4M3FN | FLOAT8_E8M0 | FLOAT8_E8M0 | null     | null   | null/FLOAT32 | FLOAT16/BFLOAT16/FLOAT32    |

- Relationship among the x1 data type, x2 data type, x1, x2, x1Scale, x2Scale, and groupSize:

  | Quantization Type | x1 Data Type | x2 Data Type | x1 Shape | x2 Shape | x1Scale Shape | x2Scale Shape | Bias Shape | yScale Shape | [groupSizeM, groupSizeN, groupSizeK] | groupSize |
  |-------|--------|--------|--------|--------|-------------|-------------|------------|---------------------------------------|--|--|
  | mx full quantization | FLOAT8_E4M3FN | FLOAT8_E4M3FN | <li>Non-transposed: (batch, m, k)</li><li>Transposed: (batch, k, m)</li> | <li>Non-transposed: (batch, k, n)</li><li>Transposed: (batch, n, k)</li> | <li>Non-transposed: (m, ceil(k / 64), 2)</li><li>Transposed: (ceil(k / 64), m, 2)</li> | <li>Non-transposed: (ceil(k / 64), n, 2)</li><li>Transposed: (n, ceil(k / 64), 2)</li> | (n,) or (batch, 1, n) | null | [1, 1, 32] | 4295032864 |

- In the mx full quantization scenario, when the data types of x1 and x2 are FLOAT8_E4M3FN, the transpose attributes of x1 and x1Scale must be consistent, and the transpose attributes of x2 and x2Scale must be consistent.

The dtype and shape requirements in pseudo-quantization scenarios are as follows:

Quantization Type	x1 dtype	x2 dtype	x1Scale dtype	x2Scale dtype	bias dtype	x1 shape	x2 shape	x1Scale shape	x2Scale shape	bias shape	yScale shape	[groupSizeM, groupSizeN, groupSizeK]
mx quantization	FLOAT8_E4M3FN	FLOAT4_E2M1	FLOAT8_E8M0	FLOAT8_E8M0	null/BFLOAT16	(m, k)	(n, k)	(m, k/32)	(n, k/32)	(1, n)	null	[0, 0, 32] / [1, 1, 32]
mx quantization	FLOAT8_E4M3FN	FLOAT32	FLOAT8_E8M0	FLOAT8_E8M0	null/BFLOAT16	(m, k)	(n, k/8)	(m, k/32)	(n, k/32)	(1, n)	null	[0, 0, 32] / [1, 1, 32]
T-CG quantization	FLOAT8_E4M3FN	FLOAT4_E2M1	null	BFLOAT16	null	(m, k)	(k, n)	null	(k/32, n)	null	(1, n)	[0, 0, 32] / [1, 1, 32]
T-CG quantization	FLOAT8_E4M3FN	FLOAT32	null	BFLOAT16	null	(m, k)	(k, n/8)	null	(k/32, n)	null	(1, n)	[0, 0, 32] / [1, 1, 32]

Constraints:
- k and n must be 64-aligned.
- When x1 is FLOAT8_E4M3FN and x2 is FLOAT32, x2 represents a densely packed data format where one FLOAT32 stores eight FLOAT4_E2M1 values.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

The following is sample code for the scenario where x2 is in NZ format (transposeX2=false).

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_permute.h"
#include "aclnnop/aclnn_quant_matmul_weight_nz.h"
#include "aclnnop/aclnn_trans_matmul_weight.h"
#include "aclnnop/aclnn_trans_quant_param_v2.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t> &shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to the device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

template <typename T>
int CreateAclTensorX2(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                      aclDataType dataType, aclTensor **tensor)
{
    auto size = static_cast<uint64_t>(GetShapeSize(shape));

    const aclIntArray *mat2Size = aclCreateIntArray(shape.data(), shape.size());
    auto ret = aclnnCalculateMatmulWeightSizeV2(mat2Size, dataType, &size);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSizeV2 failed. ERROR: %d\n", ret);
              return ret);
    size *= sizeof(T);

    // Call aclrtMalloc to allocate memory on the device side.
    ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy data from the host side to the device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    std::vector<int64_t> storageShape;
    storageShape.push_back(GetShapeSize(shape));

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              storageShape.data(), storageShape.size(), *deviceAddr);
    return 0;
}

int aclnnQuantMatmulWeightNzTest(int32_t deviceId, aclrtStream &stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Customize the construction based on the API.
    std::vector<int64_t> x1Shape = {5, 32};
    std::vector<int64_t> x2Shape = {32, 32};
    std::vector<int64_t> biasShape = {32};
    std::vector<int64_t> offsetShape = {32};
    std::vector<int64_t> scaleShape = {32};
    std::vector<int64_t> outShape = {5, 32};
    void *x1DeviceAddr = nullptr;
    void *x2DeviceAddr = nullptr;
    void *scaleDeviceAddr = nullptr;
    void *quantParamDeviceAddr = nullptr;
    void *offsetDeviceAddr = nullptr;
    void *biasDeviceAddr = nullptr;
    void *outDeviceAddr = nullptr;
    aclTensor *x1 = nullptr;
    aclTensor *x2 = nullptr;
    aclTensor *bias = nullptr;
    aclTensor *scale = nullptr;
    aclTensor *quantParam = nullptr;
    aclTensor *offset = nullptr;
    aclTensor *out = nullptr;
    std::vector<int8_t> x1HostData(5 * 32, 1);
    std::vector<int8_t> x2HostData(32 * 32, 1);
    std::vector<int32_t> biasHostData(32, 1);
    std::vector<float> scaleHostData(32, 1);
    std::vector<float> offsetHostData(32, 1);
    std::vector<uint16_t> outHostData(5 * 32, 1);  // Actually in float16 half-precision format.
    // Create the x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2 aclTensor in NZ format.
    ret = CreateAclTensorX2(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2HPTensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2HPDeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the scale aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the quantParam aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam,
                                                                                      aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the offset aclTensor.
    ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> offsetTensorPtr(offset, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> offsetDeviceAddrPtr(offsetDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = false;

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor *executor;
    void *workspaceAddr = nullptr;

    // Call the first-phase API of aclnnTransMatmulWeight.
    ret = aclnnTransMatmulWeightGetWorkspaceSize(x2, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeightGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrTrans(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrTrans.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnTransMatmulWeight.
    ret = aclnnTransMatmulWeight(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeight failed. ERROR: %d\n", ret); return ret);

    // Call the aclnn API of the TransQuantParamV2 operator in advance for the FLOAT data type scale.
    // Call the first-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Allocate device memory based on the workspaceSize calculated by the first-phase API.
    workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrV2.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret);

    // Call the first-phase API of aclnnQuantMatmulWeightNz.
    workspaceSize = 0;
    ret = aclnnQuantMatmulWeightNzGetWorkspaceSize(x1, x2, nullptr, quantParam, nullptr, nullptr, nullptr, nullptr,
                                                  bias, transposeX1, transposeX2, 0, out, &workspaceSize,
                                                  &executor);

    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Allocate device memory based on the workspaceSize calculated by the first-phase API.

    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrNZ(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrNZ.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulWeightNz.
    ret = aclnnQuantMatmulWeightNz(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNz failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0);  // In C language, fp16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 through binary conversion.
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                      size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulWeightNzTest(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzTest failed. ERROR: %d\n", ret);
                  return ret);

    Finalize(deviceId, stream);
    return 0;
}

Atlas inference products:

The sample code for the scenario where x2 is in NZ format is as follows (transposeX2=true).

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_permute.h"
#include "aclnnop/aclnn_quant_matmul_weight_nz.h"
#include "aclnnop/aclnn_trans_matmul_weight.h"
#include "aclnnop/aclnn_trans_quant_param_v2.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t> &shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Copy the host-side data to the device-side memory by calling aclrtMemcpy.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Create an aclTensor by calling the aclCreateTensor API.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

template <typename T>
int CreateAclTensorX2(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                      aclDataType dataType, aclTensor **tensor)
{
    auto size = static_cast<uint64_t>(GetShapeSize(shape));

    const aclIntArray *mat2Size = aclCreateIntArray(shape.data(), shape.size());
    auto ret = aclnnCalculateMatmulWeightSizeV2(mat2Size, dataType, &size);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSizeV2 failed. ERROR: %d\n", ret);
              return ret);
    size *= sizeof(T);

    // Apply for device-side memory by calling aclrtMalloc.
    ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Copy the host-side data to the device-side memory by calling aclrtMemcpy.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    std::vector<int64_t> storageShape;
    storageShape.push_back(GetShapeSize(shape));

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              storageShape.data(), storageShape.size(), *deviceAddr);
    return 0;
}

int aclnnQuantMatmulWeightNzTest(int32_t deviceId, aclrtStream &stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the inputs and outputs. Custom construction is required based on the API.
    std::vector<int64_t> x1Shape = {5, 32};
    std::vector<int64_t> x2Shape = {32, 32};
    std::vector<int64_t> x2TransposedShape = {32, 32};
    std::vector<int64_t> biasShape = {32};
    std::vector<int64_t> offsetShape = {32};
    std::vector<int64_t> scaleShape = {32};
    std::vector<int64_t> outShape = {5, 32};
    void *x1DeviceAddr = nullptr;
    void *x2DeviceAddr = nullptr;
    void *x2TransposedDeviceAddr = nullptr;
    void *scaleDeviceAddr = nullptr;
    void *quantParamDeviceAddr = nullptr;
    void *offsetDeviceAddr = nullptr;
    void *biasDeviceAddr = nullptr;
    void *outDeviceAddr = nullptr;
    aclTensor *x1 = nullptr;
    aclTensor *x2 = nullptr;
    aclTensor *x2Transposed = nullptr;
    aclTensor *bias = nullptr;
    aclTensor *scale = nullptr;
    aclTensor *quantParam = nullptr;
    aclTensor *offset = nullptr;
    aclTensor *out = nullptr;
    std::vector<int8_t> x1HostData(5 * 32, 1);
    std::vector<int8_t> x2HostData(32 * 32, 1);
    std::vector<int8_t> x2TransposedHostData(32 * 32, 1);
    std::vector<int32_t> biasHostData(32, 1);
    std::vector<float> scaleHostData(32, 1);
    std::vector<float> offsetHostData(32, 1);
    std::vector<uint16_t> outHostData(5 * 32, 1);  // Actually in float16 half-precision format.
    // Create the x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2 aclTensor in NZ format.
    ret = CreateAclTensorX2(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2HPTensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2HPDeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2Transposed aclTensor in NZ format.
    ret = CreateAclTensorX2(x2TransposedHostData, x2TransposedShape, &x2TransposedDeviceAddr,
                            aclDataType::ACL_INT8, &x2Transposed);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TransposedHPTensorPtr(x2Transposed,
                                                                                          aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2TransposedHPDeviceAddrPtr(x2TransposedDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the scale aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the quantParam aclTensor.
    ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam,
                                                                                      aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the offset aclTensor.
    ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> offsetTensorPtr(offset, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> offsetDeviceAddrPtr(offsetDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = true;

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor *executor;
    void *workspaceAddr = nullptr;

    // The shape of x2 needs to be transposed to nk format before transdata.
    std::vector<int64_t> dimsData = {1, 0};
    // Create the dims aclIntArray.
    aclIntArray *dims = aclCreateIntArray(dimsData.data(), dimsData.size());
    // Call the first-phase API of aclnnPermute.
    ret = aclnnPermuteGetWorkspaceSize(x2, dims, x2Transposed, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnPermuteGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrPermute(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrPermute.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnPermute.
    ret = aclnnPermute(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnPermuteGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);

    workspaceSize = 0;
    // Call the first-phase API of aclnnTransMatmulWeight.
    ret = aclnnTransMatmulWeightGetWorkspaceSize(x2Transposed, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeightGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrTrans(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrTrans.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnTransMatmulWeight.
    ret = aclnnTransMatmulWeight(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeight failed. ERROR: %d\n", ret); return ret);

    // For scales of the FLOAT data type, call the aclnn API of the TransQuantParamV2 operator in advance.
    // Call the first-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Allocate device memory based on the workspaceSize calculated by the first-phase API.
    workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrV2.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret);

    // Call the first-phase API of aclnnQuantMatmulWeightNz.
    workspaceSize = 0;
    ret = aclnnQuantMatmulWeightNzGetWorkspaceSize(x1, x2Transposed, nullptr, quantParam, nullptr, nullptr,
                                                  nullptr, nullptr, bias, transposeX1, transposeX2, 0, out,
                                                  &workspaceSize, &executor);

    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.

    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrNZ(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtrNZ.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulWeightNz.
    ret = aclnnQuantMatmulWeightNz(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNz failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0);  // fp16 data cannot be directly printed in C. Read it as uint16 and manually convert the binary to fp16.
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                      size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulWeightNzTest(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzTest failed. ERROR: %d\n", ret);
                  return ret);

    Finalize(deviceId, stream);
    return 0;
}

Atlas A3 training products:

The following is the sample code for the scenario where x2 is in NZ format (transposeX2=true).

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_cast.h"
#include "aclnnop/aclnn_npu_format_cast.h"
#include "aclnnop/aclnn_quant_matmul_weight_nz.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate: resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to the device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

// Convert the uint16_t representation of bfloat16 to float representation.
float Bf16ToFloat(uint16_t h)
{
    uint32_t sign = (h & 0x8000U) ? 0x80000000U : 0x00000000U; // sign bit
    uint32_t exponent = (h >> 7) & 0x00FFU;                    // exponent bits
    uint32_t mantissa = h & 0x007FU;                           // mantissa bits
    // The exponent bias remains unchanged.
    // Shift the mantissa left by 23 - 7, and pad the remaining bits with zeros.
    uint32_t fBits = sign | (exponent << 23) | (mantissa << (23 - 7));
    // Force cast to float.
    return *reinterpret_cast<float*>(&fBits);
}

template <typename T>
int CreateAclTensorWithFormat(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, int64_t** storageShape,
    uint64_t* storageShapeSize, void** deviceAddr, aclDataType dataType, aclTensor** tensor, aclFormat format)
{
    auto size = hostData.size() * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy data from the host to the device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, format, *storageShape, *storageShapeSize, *deviceAddr);
    return 0;
}

int aclnnQuantMatmulWeightNzTest(int32_t deviceId, aclrtStream& stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Customize the construction based on the API interface.
    int64_t m = 5;
    int64_t k = 64;
    int64_t n = 128;
    bool transposeX1 = false;
    bool transposeX2 = true;
    int64_t groupSize = 32;
    std::vector<int64_t> x1Shape = {m, k};
    std::vector<int64_t> x2Shape = {n, k};
    std::vector<int64_t> x1ScaleShape = {m, k / groupSize / 2, 2};
    std::vector<int64_t> x2ScaleShape = {n, k / groupSize / 2, 2};
    std::vector<int64_t> outShape = {m, n};
    void* x1DeviceAddr = nullptr;
    void* x2DeviceAddr = nullptr;
    void* x2NzDeviceAddr = nullptr;
    void* x2NzFp4DeviceAddr = nullptr;
    void* x1ScaleDeviceAddr = nullptr;
    void* x2ScaleDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    aclTensor* x1 = nullptr;
    aclTensor* x2 = nullptr;
    aclTensor* x1Scale = nullptr;
    aclTensor* x2Scale = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    std::vector<uint8_t> x1HostData(m * k, 0b00111000);                  // 1.0 in float8_e4m3.
    std::vector<float> x2HostData(n * k, 1);                             // The input is fp32, converted to Nz format and then cast to fp4.
    std::vector<uint8_t> x1ScaleHostData(m * k / groupSize, 0b01111111); // 1.0 in float8_e8m0.
    std::vector<uint8_t> x2ScaleHostData(n * k / groupSize, 0b10000101); // 1.0 * 64 in float8_e8m0. According to the documentation, the input needs to be scaled up by a factor of 64.
    std::vector<uint16_t> outHostData(m * k, 0);                         // Actually bfloat16.
    std::vector<int32_t> x2NzHostData(k * n, 0);
    std::vector<int32_t> x2NzFp4HostData(k * n, 0);
    int64_t* dstShape = nullptr;
    uint64_t dstShapeSize = 0;
    void* dstDeviceAddr = nullptr;
    aclTensor* x2Nz = nullptr;
    aclTensor* x2NzFp4 = nullptr;
    int actualFormat;

    // Create the x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_FLOAT8_E4M3FN, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2 aclTensor.
    ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_FLOAT, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x1Scale aclTensor.
    ret = CreateAclTensor(x1ScaleHostData, x1ScaleShape, &x1ScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, &x1Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1ScaleTensorPtr(x1Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1ScaleDeviceAddrPtr(x1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2Scale aclTensor.
    ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, &x2Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2ScaleTensorPtr(x2Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_BF16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor = nullptr;
    // Convert x2 to Nz format.
    // Calculate the shape and format of the target tensor.
    aclDataType srcDtype = aclDataType::ACL_FLOAT;
    aclDataType additionalDtype = aclDataType::ACL_FLOAT;
    ret = aclnnNpuFormatCastCalculateSizeAndFormat(x2, 29, additionalDtype, &dstShape, &dstShapeSize, &actualFormat);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCastCalculateSizeAndFormat failed. ERROR: %d\n", ret);
              return ret);

    ret = CreateAclTensorWithFormat(
        x2NzHostData, x2Shape, &dstShape, &dstShapeSize, &x2NzDeviceAddr, srcDtype, &x2Nz,
        static_cast<aclFormat>(actualFormat));
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2NzTensorPtr(x2Nz, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2NzDeviceAddrPtr(x2NzDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("CreateAclTensorWithFormat failed. ERROR: %d\n", ret); return ret);

    ret = CreateAclTensorWithFormat(
        x2NzFp4HostData, x2Shape, &dstShape, &dstShapeSize, &x2NzFp4DeviceAddr, aclDataType::ACL_FLOAT4_E2M1, &x2NzFp4,
        static_cast<aclFormat>(actualFormat));
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2NzFp4TensorPtr(x2NzFp4, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2NzFp4DeviceAddrPtr(x2NzFp4DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("CreateAclTensorWithFormat failed. ERROR: %d\n", ret); return ret);

    // Call the first-phase API of aclnnNpuFormatCastGetWorkspaceSize.
    ret = aclnnNpuFormatCastGetWorkspaceSize(x2, x2Nz, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceNzAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceNzAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceNzAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceNzAddrPtr.reset(workspaceNzAddr);
    }

    // Call the second-phase API of aclnnNpuFormatCastGetWorkspaceSize.
    ret = aclnnNpuFormatCast(workspaceNzAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCast failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to finish.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // Call cast to convert x2 from fp32 to fp4_e2m1.
    ret = aclnnCastGetWorkspaceSize(x2Nz, aclDataType::ACL_FLOAT4_E2M1, x2NzFp4, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize0 failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceCastAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceCastAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceCastAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceCastAddrPtr.reset(workspaceCastAddr);
    }
    ret = aclnnCast(workspaceCastAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast0 failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // Call the first-phase API of aclnnQuantMatmulWeightNz.
    workspaceSize = 0;
    executor = nullptr;
    ret = aclnnQuantMatmulWeightNzGetWorkspaceSize(
        x1, x2NzFp4, x1Scale, x2Scale, nullptr, nullptr, nullptr, nullptr, bias, transposeX1, transposeX2, groupSize,
        out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulWeightNz.
    ret = aclnnQuantMatmulWeightNz(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNz failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from device memory to host memory. Modify based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0); // In C, fp16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 via binary representation.
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %.1f\n", i, Bf16ToFloat(resultData[i]));
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Enter your actual device ID for deviceId.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulWeightNzTest(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzTest failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}