aclnnDualLevelQuantMatmulWeightNz

Applicable Products

Function

• Description: Completes the second-level quantization mxfp4 matrix multiplication calculation, where the x2 parameter must be in NZ format. You can obtain the NZ format by processing x2 in ND format through aclnnTransMatmulWeight or aclnnNpuFormatCast.

• Formula

  $$out =\sum_{i}^{level0GroupSize} x1Level0Scale @ x2Level0Scale \sum_{ij}^{level1GroupSize} ((x1Level1Scale @ x1_{ij})@ (x2Level1Scale @ x2_{ij})) + bias$$

  • x1 and x2 are the left and right matrices for matrix multiplication, with the data type FLOAT4_E2M1.

  • x1Level0Scale and x2Level0Scale: first-level quantization parameters, with the data type FLOAT32.

  • x1Level1Scale and x2Level1Scale: second-level quantization parameters, with the data type FLOAT8_E8M0.

  • bias: optional parameter, the bias accumulated after the matrix multiplication operation, with the data type FLOAT32.

  • level0GroupSize: the size of the first-level quantization groupsize, which only supports 512.

  • level1GroupSize: the size of the first-level quantization groupsize, which only supports 32.

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that contains the operator computation process, and then call the "aclnnDualLevelQuantMatmulWeightNz" API to perform the computation.

aclnnStatus aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize(
    const aclTensor* x1,
    const aclTensor* x2,
    const aclTensor* x1Level0Scale,
    const aclTensor* x2Level0Scale,
    const aclTensor* x1Level1Scale,
    const aclTensor* x2Level1Scale,
    const aclTensor* optionalBias,
    bool transposeX1,
    bool transposeX2,
    int64_t level0GroupSize,
    int64_t level1GroupSize,
    aclTensor* out,
    uint64_t* workspaceSize,
    aclOpExecutor** executor)

aclnnStatus aclnnDualLevelQuantMatmulWeightNz(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize

• Parameters

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  x1(aclTensor*)	Input	The left matrix in the matrix multiplication operation.	Empty tensors are not supported. Only non-transposed format is supported.	FLOAT4_E2M1	ND	2	-
  x2(aclTensor*)	Input	Right matrix in the matrix multiplication operation.	Empty tensors are not supported. Only transpose is supported.	FLOAT4_E2M1	FRACTAL_NZ	4	√
  x1Level0Scale(aclTensor*)	Input	Scale factor of the first-level quantization parameter of x1, corresponding to x1Level0Scale in the formula.	Empty tensor is not supported. Only non-transposed format is supported.	FLOAT32	ND	2	-
  x1Level1Scale(aclTensor*)	Input	Scale factor for the second-level quantization parameter of x1, corresponding to x1Level1Scale in the formula.	Empty tensors are not supported. Only non-transposed format is supported.	FLOAT8_E8M0	ND	3	-
  x2Level0Scale(aclTensor*)	Input	Scale factor of the first-level quantization parameter of x2, corresponding to x2Level0Scale in the formula.	Null tensors are not supported. Only non-transposed tensors are supported.	FLOAT32	ND	2	-
  x2Level1Scale(aclTensor*)	Input	Scale factor of the second-level quantization parameter of x2, corresponding to x2Level1Scale in the formula.	Empty tensors are not supported. Only transpose is supported.	FLOAT8_E8M0	ND	3	-
  optionalBias(aclTensor*)	Optional input	Bias accumulated after the matrix multiplication operation, corresponding to bias in the corresponding formula.	Empty tensors are not supported.	FLOAT32	ND	1	-
  transposeX1(bool)	Input	Indicates whether to transpose the input shape of x1.	-	BOOL	-	-	-
  transposeX2(bool)	Input	Indicates whether the input shape of x2 is transposed.	-	BOOL	-	-	-
  level0GroupSize(int64_t)	Input	Specifies the groupSize input for dequantization computation on x1 and x2 under first-level quantization, describing the size of the data to be dequantized corresponding to a group of dequantization parameters in the reduce direction.	First-level quantization groupsize, only 512 is supported.	INT64	-	-	-
  level1GroupSize(int64_t)	Input	Specifies the groupSize input for dequantization computation on x1 and x2 under second-level quantization, describing the size of the data to be dequantized corresponding to a set of dequantization parameters in the Reduce direction.	Second-level quantization groupsize, only 32 is supported.	INT64	-	-	-
  out(aclTensor)	Output	Output of the matrix multiplication computation.	Empty tensors are not supported.	FLOAT16, BFLOAT16	ND	2	✓
  workspaceSize(uint64_t)	Output	Returns the workspace size to be allocated on the Device side.	-	-	-	-	-
  executor(aclOpExecutor)	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The input x1, x2, x1Level0Scale, x1Level1Scale, x2Level0Scale, x2Level1Scale, or out is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	x1, x2, x1Level0Scale, x1Level1Scale, x2Level0Scale, x2Level1Scale, level0GroupSize, or level1GroupSize is a null tensor.
  		The data type or data format of x1, x2, x1Level0Scale, x1Level1Scale, x2Level0Scale, x2Level1Scale, level0GroupSize, level1GroupSize, or out is not within the supported range.
  		The shape of x1, x2, x1Level0Scale, x1Level1Scale, x2Level0Scale, x2Level1Scale, level0GroupSize, level1GroupSize, or out does not meet the validation conditions.
  		The input level0GroupSize or level1GroupSize does not meet the validation conditions.

aclnnDualLevelQuantMatmulWeightNz

• Parameters

  Parameter	Input/Output	Description
  workspace	Input	Memory address of the workspace applied for on the Device side.
  workspaceSize	Input	Size of the workspace applied for on the Device side, obtained by the first-phase API aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation flow.
  stream	Input	Specifies the stream for executing the task.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Deterministic computation: aclnnDualLevelQuantMatmulWeightNz defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_dual_level_quant_matmul_nz.h"
#include "aclnnop/aclnn_npu_format_cast.h"
#include "aclnnop/aclnn_cast.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

template <typename T>
void PrintMat(std::vector<T> resultData, std::vector<int64_t> resultShape)
{
    int64_t m = resultShape[0];
    int64_t n = resultShape[1];
    for (size_t i = 0; i < m; i++) {
        printf(i == 0 ? "[[" : " [");
        for (size_t j = 0; j < n; j++) {
            std::cout << resultData[i * n + j] << (j == n - 1 ? "" : ", ");
            if (j == 2 && j + 3 < n) {
                printf("..., ");
                j = n - 4;
            }
        }
        printf(i < m - 1 ? "],\n" : "]]\n");
        if (i == 2 && i + 3 < m) {
            printf(" ... \n");
            i = m - 4;
        }
    }
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, const int64_t* storageShape,
    int64_t storageShapeSize, void** deviceAddr, aclDataType dataType, aclTensor** tensor,
    aclFormat format = aclFormat::ACL_FORMAT_ND)
{
    auto size = hostData.size() * sizeof(T);
    // Call aclrtMalloc to allocate device memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host data to device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, format, storageShape, storageShapeSize, *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int AclnnDualLevelQuantMatmulWeightNz(int32_t deviceId, aclrtStream stream)
{
    int ret = 0;
    // 2. Construct the inputs and outputs. Customize the construction based on the API interface.
    constexpr int64_t B4_IN_B8_NUMS = 2L;
    constexpr int64_t B8_IN_B16_NUMS = 2L;
    int64_t m = 256;
    int64_t k = 1024;
    int64_t n = 512;
    int64_t level0GroupSize = 512;
    int64_t level1GroupSize = 32;
    bool transposeX1 = false;
    bool transposeX2 = true;
    std::vector<int64_t> x1Shape = {m, k};
    std::vector<int64_t> x2Shape = {n, k};
    std::vector<int64_t> biasShape = {n};
    std::vector<int64_t> x1Level0ScaleShape = {m, k / level0GroupSize};
    std::vector<int64_t> x1Level1ScaleShape = {m, k / level1GroupSize / B8_IN_B16_NUMS, B8_IN_B16_NUMS};
    std::vector<int64_t> x2Level0ScaleShape = {k / level0GroupSize, n};
    std::vector<int64_t> x2Level1ScaleShape = {n, k / level1GroupSize / B8_IN_B16_NUMS, B8_IN_B16_NUMS};
    std::vector<int64_t> outShape = {m, n};

    void* x1DeviceAddr = nullptr;
    void* x2DeviceAddr = nullptr;
    void* x2NzDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* x1Level0ScaleDeviceAddr = nullptr;
    void* x1Level1ScaleDeviceAddr = nullptr;
    void* x2Level0ScaleDeviceAddr = nullptr;
    void* x2Level1ScaleDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* outFp32DeviceAddr = nullptr;

    aclTensor* x1 = nullptr;
    aclTensor* x2 = nullptr;
    aclTensor* x2Nz = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* x1Level0Scale = nullptr;
    aclTensor* x1Level1Scale = nullptr;
    aclTensor* x2Level0Scale = nullptr;
    aclTensor* x2Level1Scale = nullptr;
    aclTensor* out = nullptr;
    aclTensor* outFp32 = nullptr;

    // Binary representation of 1.0 in fp4, 0b0010. Each uint8_t represents two fp4 values, using uint8_t to carry fp4.
    std::vector<uint8_t> x1HostData(GetShapeSize(x1Shape) / 2, 0b0010'0010);
    std::vector<uint8_t> x2HostData(GetShapeSize(x2Shape), 0b0010'0010);
    std::vector<float> biasHostData(n, 1002.0f);
    std::vector<float> x1Level0ScaleHostData(GetShapeSize(x1Level0ScaleShape), 1.0f);
    std::vector<float> x2Level0ScaleHostData(GetShapeSize(x2Level0ScaleShape), 1.0f);
    // Binary representation of 1.0 in fp8e8m0, 0b0111'1111.
    std::vector<uint8_t> x1Level1ScaleHostData(GetShapeSize(x1Level1ScaleShape), 0b0111'1111);
    std::vector<uint8_t> x2Level1ScaleHostData(GetShapeSize(x2Level1ScaleShape), 0b0111'1111);
    // Fill the output with zeros.
    std::vector<uint16_t> outHostData(GetShapeSize(outShape), 0);
    std::vector<float> outFp32HostData(GetShapeSize(outShape), 0.0f);

    // Create the x1 aclTensor.
    ret = CreateAclTensor(
        x1HostData, x1Shape, x1Shape.data(), x1Shape.size(), &x1DeviceAddr, aclDataType::ACL_FLOAT4_E2M1, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the x2 aclTensor.
    // NpuFormatCast cannot convert the B4 type, so B8 is used instead, halving the inner axis.
    x2Shape[1] /= 2;
    ret =
        CreateAclTensor(x2HostData, x2Shape, x2Shape.data(), x2Shape.size(), &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Use the NpuFormatCast API to convert to the actual tensor, or directly construct the weightNZ data.
    int64_t* x2NzShape = nullptr;
    uint64_t x2NzShapeSize = 0;
    int x2NzFormat;
    // Calculate the shape and format of the target tensor.
    ret = aclnnNpuFormatCastCalculateSizeAndFormat(
        x2, static_cast<int>(aclFormat::ACL_FORMAT_FRACTAL_NZ), static_cast<int>(aclDataType::ACL_INT8), &x2NzShape,
        &x2NzShapeSize, &x2NzFormat);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCastCalculateSizeAndFormat failed. ERROR: %d\n", ret);
              return ret);
    ret = CreateAclTensor(
        x2HostData, x2Shape, x2NzShape, x2NzShapeSize, &x2NzDeviceAddr, aclDataType::ACL_INT8, &x2Nz,
        static_cast<aclFormat>(x2NzFormat));
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2NzTensorPtr(x2Nz, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2NzDeviceAddrPtr(x2NzDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("CreateAclTensorWithFormat failed. ERROR: %d\n", ret); return ret);

    // Create the x1Level0Scale aclTensor.
    ret = CreateAclTensor(
        x1Level0ScaleHostData, x1Level0ScaleShape, x1Level0ScaleShape.data(), x1Level0ScaleShape.size(),
        &x1Level0ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x1Level0Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1Level0ScaleTensorPtr(
        x1Level0Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1Level0ScaleDeviceAddrPtr(x1Level0ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the x1Level1Scale aclTensor.
    ret = CreateAclTensor(
        x1Level1ScaleHostData, x1Level1ScaleShape, x1Level1ScaleShape.data(), x1Level1ScaleShape.size(),
        &x1Level1ScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, &x1Level1Scale, aclFormat::ACL_FORMAT_NCL);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1Level1ScaleTensorPtr(
        x1Level1Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1Level1ScaleDeviceAddrPtr(x1Level1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the x2Level0Scale aclTensor.
    ret = CreateAclTensor(
        x2Level0ScaleHostData, x2Level0ScaleShape, x2Level0ScaleShape.data(), x2Level0ScaleShape.size(),
        &x2Level0ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x2Level0Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2Level0ScaleTensorPtr(
        x2Level0Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2Level0ScaleDeviceAddrPtr(x2Level0ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the x2Level1Scale aclTensor.
    ret = CreateAclTensor(
        x2Level1ScaleHostData, x2Level1ScaleShape, x2Level1ScaleShape.data(), x2Level1ScaleShape.size(),
        &x2Level1ScaleDeviceAddr, aclDataType::ACL_FLOAT8_E8M0, &x2Level1Scale, aclFormat::ACL_FORMAT_NCL);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2Level1ScaleTensorPtr(
        x2Level1Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2Level1ScaleDeviceAddrPtr(x2Level1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the bias aclTensor.
    ret = CreateAclTensor(
        biasHostData, biasShape, biasShape.data(), biasShape.size(), &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> biasTensorPtr(bias, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // Create the out aclTensor.
    ret = CreateAclTensor(
        outHostData, outShape, outShape.data(), outShape.size(), &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the outFp32 aclTensor.
    ret = CreateAclTensor(
        outFp32HostData, outShape, outShape.data(), outShape.size(), &outFp32DeviceAddr, aclDataType::ACL_FLOAT, &outFp32);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outFp32TensorPtr(outFp32, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outFp32DeviceAddrPtr(outFp32DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor = nullptr;
    void* workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceAddrPtr(nullptr, aclrtFree);

    // First, use NpuFormatCast to convert x2 to NZ format.
    // Call the first-phase API of aclnnNpuFormatCastGetWorkspaceSize.
    ret = aclnnNpuFormatCastGetWorkspaceSize(x2, x2Nz, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    void* workspaceNpuFormatCastAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceNpuFormatCastAddrPtr(nullptr, aclrtFree);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceNpuFormatCastAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceNpuFormatCastAddrPtr.reset(workspaceNpuFormatCastAddr);
    }
    // Call the second-phase API of aclnnNpuFormatCastGetWorkspaceSize.
    ret = aclnnNpuFormatCast(workspaceNpuFormatCastAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnNpuFormatCast failed. ERROR: %d\n", ret); return ret);

    // Call the first-phase API of aclnnDualLevelQuantMatmulWeightNz.
    ret = aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize(
        x1, x2Nz, x1Level0Scale, x2Level0Scale, x1Level1Scale, x2Level1Scale, bias, transposeX1, transposeX2,
        level0GroupSize, level1GroupSize, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS,
              LOG_PRINT("aclnnDualLevelQuantMatmulWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnDualLevelQuantMatmulWeightNz.
    ret = aclnnDualLevelQuantMatmulWeightNz(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnDualLevelQuantMatmulWeightNz failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value, convert the result on the device-side memory to FP32 and copy it to the host side. Modify based on the specific API definition.
    ret = aclnnCastGetWorkspaceSize(out, aclDataType::ACL_FLOAT, outFp32, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    ret = aclnnCast(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast failed. ERROR: %d\n", ret); return ret);

    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 6. Obtain the output value, and copy the result on the device-side memory to the host side. Modify based on the specific API definition.
    auto size = GetShapeSize(outShape);
    ret = aclrtMemcpy(
        outFp32HostData.data(), size * sizeof(outFp32HostData[0]), outFp32DeviceAddr, size * sizeof(outFp32HostData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    PrintMat(outFp32HostData, outShape);
    return ret;
}

int main()
{
    // 1. (Boilerplate) Initialize device/stream. See the acl API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
    // Execute the test process and release memory.
    AclnnDualLevelQuantMatmulWeightNz(deviceId, stream);
    Finalize(deviceId, stream);
    return 0;
}