aclnnQuantMatmulV5

📄 View Source Code

Applicable Products

Function

• Description: Performs quantized matrix multiplication computation.

  • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

    Compatible with the API functions of aclnnQuantMatmulV3 and aclnnQuantMatmulV4. Performs quantized matrix multiplication computation. The minimum supported input dimension is 1, and the maximum supported input dimension is 2. A similar API is aclnnMm (matrix multiplication that only supports 2D tensors as input).

  • Ascend 950PR/Ascend 950DT:

    Compatible with the aclnnQuantMatmulV3 and aclnnQuantMatmulV4 APIs, and additionally supports features such as G-B, B-B, T-CG, and mxquantization mode, with newly added support for x1 and x2 inputs with dtype FLOAT8_E4M3FN, FLOAT8_E5M2, HIFLOAT8, and FLOAT4_E2M1. Performs quantized matrix multiplication, with a minimum input dimension of 2 and a maximum input dimension of 6. Similar APIs include aclnnMm (which only supports 2D tensors as input for matrix multiplication) and aclnnBatchMatMul (which only supports 3D matrix multiplication, where the first dimension is the batch dimension).

• Formula:

  • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

    Supports K-C && K-T, T-C && T-T, G-B, and K-G quantization modes. For the input/output data type combinations corresponding to different quantization modes, see Constraints.

    K-G Quantization Mode

    • x1 is INT8, x2 is INT32, x1Scale is FLOAT32, x2Scale is UINT64/INT64, and yOffset is FLOAT32:

      $$out = ((x1 @ (x2*x2Scale)) + yOffset) * x1Scale$$

    • x1 and x2 are INT4, x1Scale and x2Scale are FLOAT32, x2Offset is FLOAT16, and out is FLOAT16/BFLOAT16 (pertoken-pergroup asymmetric quantization):

      $$out = x1Scale * x2Scale * (x1 @ x2 - x1 @ x2Offset)$$
    K-C && K-T Quantization Mode

    • x1Scale without bias:

      $$out = x1@x2 * x2Scale * x1Scale$$

    • x1Scale, bias INT32 (no offset in this scenario):

      $$out = (x1@x2 + bias) * x2Scale * x1Scale$$

    • x1Scale, bias BFLOAT16/FLOAT16/FLOAT32 (no offset in this scenario):

      $$out = x1@x2 * x2Scale * x1Scale + bias$$
    T-C && T-T Quantization Mode

    • Without x1Scale and without bias:

      $$out = x1@x2 * x2Scale + x2Offset$$

    • bias INT32:

      $$out = (x1@x2 + bias) * x2Scale + x2Offset$$

    • bias BFLOAT16/FLOAT32 (no offset in this scenario):

      $$out = x1@x2 * x2Scale + bias$$
    G-B quantization mode

    • x1 and x2 are INT8, x1Scale and x2Scale are FLOAT32, bias is FLOAT32, and out is FLOAT16/BFLOAT16 (pergroup-perblock quantization):$$out = (x1 @ x2) * x1Scale * x2Scale + bias$$

  • Atlas inference products:

    Supports K-C quantization mode. For the input/output data type combinations corresponding to different quantization modes, see Constraints.

    K-C Quantization Mode

    • x1Scale without bias:

      $$out = x1@x2 * x2Scale * x1Scale$$

    • x1Scale, bias INT32 (no offset in this scenario):

      $$out = (x1@x2 + bias) * x2Scale * x1Scale$$

  • Ascend 950PR/Ascend 950DT:

    Supports T-C && T-T, K-C && K-T, G-B, B-B, mx, and T-CG quantization modes. For the input/output data type combinations corresponding to different quantization modes, see Constraints.

    T-C && T-T Quant Mode

    • x1 and x2 are INT8, no x1Scale, x2Scale is INT64/UINT64, optional parameter x2Offset is FLOAT32, and optional parameter bias is INT32:

      $$out = (x1@x2 + bias) * x2Scale + x2Offset$$

    • x1 and x2 are INT8, without x1Scale, x2Scale is INT64/UINT64, and the optional parameter bias is INT32;

      Or x1 and x2 are FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, without x1Scale, x2Scale is INT64/UINT64, and the optional parameter bias is FLOAT32:

      $$out = (x1@x2 + bias) * x2Scale$$

    • x1 and x2 are INT8, without x1Scale, x2Scale is BFLOAT16/FLOAT32, and the optional parameter bias is BFLOAT16/FLOAT32:

      $$out = x1@x2 * x2Scale + bias$$

    • x1 and x2 are FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, x1Scale is FLOAT32, x2Scale is FLOAT32, and the optional parameter bias is FLOAT32:

    $$out = (x1@x2 + bias) * x2Scale * x1Scale$$
    K-C && K-T Quantization Mode

    • x1 and x2 are INT8, x1Scale is FLOAT32, x2Scale is BFLOAT16/FLOAT32, and the optional parameter bias is INT32;

    Or x1 and x2 are FLOAT8_E4M3FN/FLOAT8_E5M2/HIFLOAT8, x1Scale is FLOAT32, x2Scale is FLOAT32, and the optional parameter bias is FLOAT32:

    $$out = (x1@x2 + bias) * x2Scale * x1Scale$$

    • x1 and x2 are INT8, x1Scale is FLOAT32, x2Scale is BFLOAT16/FLOAT32, and the optional parameter bias is BFLOAT16/FLOAT32;

    or x1 and x2 are INT8, x1Scale is FLOAT32, x2Scale is FLOAT32, and the optional parameter bias is FLOAT16/FLOAT32:

    $$out = x1@x2 * x2Scale * x1Scale + bias$$
    G-B && B-B && mx quantization mode

    $$out[m,n] = \sum_{j=0}^{kLoops-1} ((\sum_{k=0}^{gsK-1} (x1Slice * x2Slice))* (x1Scale[m/gsM, j] * x2Scale[j, n/gsN]))+bias[n]$$

    Where gsM, gsN, and gsK represent groupSizeM, groupSizeN, and groupSizeK, respectively; x1Slice represents the vector of length groupSizeK in the m-th row of x1, and x2Slice represents the vector of length groupSizeK in the n-th column of x2. The K-axis slices start from j*groupSizeK, with j in the range [0, kLoops), where kLoops = ceil(K / groupSizeK) and K is the K-axis length. The last slice is allowed to be shorter than groupSizeK. The bias is included only in the mx quantization mode. For the G-B, B-B, and mx quantization modes, the supported [groupSizeM, groupSizeN, groupSizeK] value combinations are [1, 128, 128], [128, 128, 128], and [1, 1, 32], respectively.

    T-CG Quantization Mode

    $$out = (x1@(x2 * x2Scale)) * yScale$$

Function Prototype

Each operator is divided into a two-phase API. The "aclnnQuantMatmulV5GetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnQuantMatmulV5" API is called to perform the computation.

aclnnStatus aclnnQuantMatmulV5GetWorkspaceSize(
  const aclTensor *x1,
  const aclTensor *x2,
  const aclTensor *x1Scale,
  const aclTensor *x2Scale,
  const aclTensor *yScale,
  const aclTensor *x1Offset,
  const aclTensor *x2Offset,
  const aclTensor *yOffset,
  const aclTensor *bias,
  bool             transposeX1,
  bool             transposeX2,
  int64_t          groupSize,
  aclTensor       *out,
  uint64_t        *workspaceSize,
  aclOpExecutor   **executor)

aclnnStatus aclnnQuantMatmulV5(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnQuantMatmulV5GetWorkspaceSize

• Parameters

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  x1 (aclTensor*)	Input	Input x1 in the formula.	Empty tensors are not supported. Only the last m and k axes support non-contiguous tensors when transposed. Other axes do not support non-contiguous tensors.	INT41, INT8, INT321, FLOAT8_E4M3FN2, FLOAT8_E5M22, HIFLOAT82, FLOAT4_E2M12	ND	2-6	√
  x2(aclTensor*)	Input	Input x2 in the formula.	Null tensors are not supported. In NZ format, null tensors are not supported. In NZ format, the shape supports 4 to 8 dimensions. In ND format, non-contiguous tensors with the last two axes transposed are supported. Non-contiguous tensors in other scenarios are not supported.	INT41, INT8, INT321, FLOAT8_E4M3FN2, FLOAT8_E5M22, HIFLOAT82, FLOAT4_E2M12	ND, NZ	2-8	√
  x1Scale (aclTensor*)	Optional input	Input x1Scale in the formula.	Null tensors are not supported.	FLOAT32, FLOAT8_E8M02, FLOAT8_E4M3FN2, FLOAT8_E5M22, HIFLOAT82	ND	1-6	√
  x2Scale(aclTensor*)	Input	Represents the quantization parameter, which is the input x2Scale in the formula.	Empty tensors are not supported.	UINT64, INT64, FLOAT32, BFLOAT16, FLOAT8_E8M02	ND	1-6	√
  yScale (aclTensor*)	Optional input	Dequantization scale parameter for output y.	Null tensors are not supported. If this parameter is not used, pass nullptr.	UINT642, INT642	ND	2	×
  x1Offset (aclTensor*)	Optional input	Input x1Offset in the formula.	Reserved parameter. Not supported in the current version. Pass nullptr.	-	-	-	-
  x2Offset(aclTensor*)	Optional input	Input x2Offset in the formula.	Null tensors are not supported.	FLOAT16, FLOAT32	ND	1-2	-
  yOffset (aclTensor*)	Optional input	Input yOffset in the formula.	-	FLOAT32	ND	1	-
  bias(aclTensor*)	Optional input	Input bias in the formula.	Null tensors are not supported.	INT32, FLOAT32, BFLOAT16, FLOAT16	ND	1-3	×
  transposeX1(bool)	Input	Indicates whether to transpose the input shape of x1.	-	BOOL	-	-	-
  transposeX2(bool)	Input	Specifies whether to transpose the input shape of x2.	-	BOOL	-	-	-
  groupSize(int64_t)	Optional Input	Specifies the quantization group size in the m, n, and k directions.	Consists of three values concatenated from three directions: groupSizeM, groupSizeN, and groupSizeK. Each value occupies 16 bits, occupying the lower 48 bits of the int64_t type groupSize (the upper 16 bits of groupSize are invalid). See the formula below the table.	INT64	-	-	-
  out (aclTensor)	Output	Output out in the formula.	Null tensors are not supported.	FLOAT16, INT8, BFLOAT16, INT32, FLOAT32	ND	2	-
  workspaceSize(uint64_t)	Output	Returns the workspace size to be applied for on the Device side.	-	-	-	-	-
  executor(aclOpExecutor)	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

  • Note: Optional inputs refer to optional quantization parameters, which support passing nullptr.

  • Ascend 950PR/Ascend 950DT:

    • The superscript "1" in the data type column of the table above indicates a data type not supported by this series.

    • The input parameters x1 and x2 do not support INT4 or INT32 types.

    • When x2 is in ND format, if the input parameter x1 is an empty tensor with m=0 or x2 is an empty tensor with n=0, the output is an empty tensor. When x2 is in FRACTAL_NZ format, if the input parameter x1 is an empty tensor with m=0, the output is an empty tensor.

  • Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

    • The superscript "2" in the data type column of the table above indicates a data type not supported by this series.

  • Formula:

    $$groupSize = groupSizeK | groupSizeN << 16 | groupSizeM << 32$$

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The passed x1, x2, x1Scale, x2Scale, yOffset, or out is a null pointer.
  ACLNN_ERR_PARAM_INVALID	161002	x1, x2, bias, x2Scale, x2Offset, or out is a null tensor.
  		The data type or data format of x1, x2, bias, x1Scale, x2Scale, x2Offset, or out is not within the supported range.
  		The shape of x1, x2, bias, x1Scale, x2Scale, x2Offset, or out does not meet the validation conditions.
  		The passed groupSize does not meet the validation conditions, or when the passed groupSize is 0, the groupSize cannot be inferred from the shape relationship between x1, x2 and x1Scale, x2Scale.

aclnnQuantMatmulV5

• Parameters

  Parameter	Input/Output	Description
  workspace	Input	Specifies the workspace memory address applied for on the Device side.
  workspaceSize	Input	Specifies the workspace size applied for on the Device side, obtained through the first-phase API aclnnQuantMatmulV5GetWorkspaceSize.
  executor	Input	Operator executor that contains the operator computation flow.
  stream	Input	Specifies the stream for task execution.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

• Deterministic computation:

  • aclnnQuantMatmulV5 defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

• Ascend 950PR/Ascend 950DT:

x1 is FLOAT8_E4M3FN, x2 is FLOAT4_E2M1, x2Scale is BFLOAT16, and yScale is UINT64.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul_v5.h"
#include "aclnnop/aclnn_trans_quant_param_v2.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate: resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(
    const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
    aclTensor** tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to the device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of the contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

// Convert the uint16_t representation of bfloat16 to a float representation.
float Bf16ToFloat(uint16_t h)
{
    uint32_t sign = (h & 0x8000U) ? 0x80000000U : 0x00000000U; // sign bit
    uint32_t exponent = (h >> 7) & 0x00FFU;                    // exponent bits
    uint32_t mantissa = h & 0x007FU;                           // mantissa bits
    // Keep the exponent bias unchanged.
    // Shift mantissa left by 23 - 7, and pad the remaining bits with zeros.
    uint32_t fBits = sign | (exponent << 23) | (mantissa << (23 - 7));
    // Cast to float.
    return *reinterpret_cast<float*>(&fBits);
}

int AclnnQuantMatmulV5Test(int32_t deviceId, aclrtStream& stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Customize the construction based on the API interface.
    std::vector<int64_t> x1Shape = {5, 64};
    std::vector<int64_t> x2Shape = {8, 64};
    std::vector<int64_t> x2ScaleShape = {8, 2};
    std::vector<int64_t> yScaleShape = {1, 8};
    std::vector<int64_t> outShape = {5, 8};
    void* x1DeviceAddr = nullptr;
    void* x2DeviceAddr = nullptr;
    void* x2ScaleDeviceAddr = nullptr;
    void* yScaleDeviceAddr = nullptr;
    void* quantParamDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    aclTensor* x1 = nullptr;
    aclTensor* x2 = nullptr;
    aclTensor* x2Scale = nullptr;
    aclTensor* yScale = nullptr;
    aclTensor* yOffset = nullptr;
    aclTensor* quantParam = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    std::vector<uint8_t> x1HostData(5 * 64, 0b00111000);                 // 0b00111000 represents 1.0 in fp8_e4m3fn.
    std::vector<uint8_t> x2HostData(8 * 64 / 2, 0b0010 + (0b0010 << 4)); // 0b0010 represents 1.0 in fp4_e2m1. Here uint8 is used to represent two fp4 values.
    std::vector<uint16_t> x2ScaleHostData(8 * 2, 0b0011111110000000); // 1.0 in bf16.
    std::vector<float> yScaleHostData(8, 1.0);                        // 1.0 in fp32.
    std::vector<uint64_t> quantParamHostData(8, 0);
    std::vector<uint16_t> outHostData(40, 0); // Actually bfloat16.
    // Create x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_FLOAT8_E4M3FN, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2 aclTensor.
    ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_FLOAT4_E2M1, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2Scale aclTensor.
    ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_BF16, &x2Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> x2ScaleTensorPtr(x2Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create yScale aclTensor.
    ret = CreateAclTensor(yScaleHostData, yScaleShape, &yScaleDeviceAddr, aclDataType::ACL_FLOAT, &yScale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> yScaleTensorPtr(yScale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> yScaleDeviceAddrPtr(yScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create quantParam aclTensor.
    ret = CreateAclTensor(quantParamHostData, yScaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> quantParamTensorPtr(quantParam, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the output aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_BF16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor*)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void*)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = true;
    int64_t groupSize = 32;

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor = nullptr;

    // Call the first-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2GetWorkspaceSize(yScale, yOffset, quantParam, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceQuantParamAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceQuantParamAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceQuantParamAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceQuantParamAddrPtr.reset(workspaceQuantParamAddr);
    }
    // Call the second-phase API of aclnnTransQuantParamV2.
    ret = aclnnTransQuantParamV2(workspaceQuantParamAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret);

    workspaceSize = 0;
    executor = nullptr;
    // Call the first-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5GetWorkspaceSize(
        x1, x2, nullptr, x2Scale, quantParam, nullptr, nullptr, nullptr, bias, transposeX1, transposeX2, groupSize, out,
        &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void*)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this step based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0); // In C language, fp16 data cannot be printed directly. It needs to be read as uint16 and manually converted to fp16 via binary.
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %.1f\n", i, Bf16ToFloat(resultData[i]));
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize device/stream. Refer to the ACL API manual.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = AclnnQuantMatmulV5Test(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("AclnnQuantMatmulV5Test failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}

• Ascend 950PR/Ascend 950DT:

x1 is INT8, x2 is INT8, x1Scale is FLOAT32, x2Scale is FLOAT32, and bias is INT32.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul_v5.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

    int64_t
    GetShapeSize(const std::vector<int64_t> &shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream)
{
    // Boilerplate, resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to the device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of the contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int aclnnQuantMatmulV5Test(int32_t deviceId, aclrtStream &stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs based on the API's interface definition.
    std::vector<int64_t> x1Shape = {5, 16};
    std::vector<int64_t> x2Shape = {16, 8};
    std::vector<int64_t> biasShape = {8};
    std::vector<int64_t> x2OffsetShape = {8};
    std::vector<int64_t> x1ScaleShape = {5};
    std::vector<int64_t> x2ScaleShape = {8};
    std::vector<int64_t> outShape = {5, 8};
    void *x1DeviceAddr = nullptr;
    void *x2DeviceAddr = nullptr;
    void *x2ScaleDeviceAddr = nullptr;
    void *x2OffsetDeviceAddr = nullptr;
    void *x1ScaleDeviceAddr = nullptr;
    void *biasDeviceAddr = nullptr;
    void *outDeviceAddr = nullptr;
    aclTensor *x1 = nullptr;
    aclTensor *x2 = nullptr;
    aclTensor *bias = nullptr;
    aclTensor *x2Scale = nullptr;
    aclTensor *x2Offset = nullptr;
    aclTensor *x1Scale = nullptr;
    aclTensor *out = nullptr;
    std::vector<int8_t> x1HostData(80, 1);
    std::vector<int8_t> x2HostData(128, 1);
    std::vector<int32_t> biasHostData(8, 1);
    std::vector<float> x2ScaleHostData(8, 1);
    std::vector<float> x2OffsetHostData(8, 1);
    std::vector<float> x1ScaleHostData(5, 1);
    std::vector<uint16_t> outHostData(40, 1);  // Actually in float16 half-precision format.
    // Create the x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2 aclTensor.
    ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x1Scale aclTensor.
    ret = CreateAclTensor(x1ScaleHostData, x1ScaleShape, &x1ScaleDeviceAddr,
                          aclDataType::ACL_FLOAT, &x1Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1ScaleTensorPtr(x1Scale,
                                                                                          aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1ScaleDeviceAddrPtr(x1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the x2Scale aclTensor.
    ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x2Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(x2Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create the out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = false;

    // 3. Call the CANN operator library API. Replace it with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor *executor = nullptr;
    // Call the first-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5GetWorkspaceSize(x1, x2, x1Scale, x2Scale, nullptr, nullptr, nullptr, nullptr, bias,
                                             transposeX1, transposeX2, 0, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5GetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void *workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronize and wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from device memory to host memory. Modify this based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0);  // In C, fp16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 through binary conversion.
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                      size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. Refer to the acl API manual.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulV5Test(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5Test failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}

• Ascend 950PR/Ascend 950DT:

x1 and x2 are FLOAT8_E4M3FN, x1Scale is FLOAT32, x2Scale is FLOAT32, no x2Offset, and bias is FLOAT32.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul_v5.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

    int64_t
    GetShapeSize(const std::vector<int64_t> &shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream)
{
    // Boilerplate: resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host data to device memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Calculate the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                              shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int aclnnQuantMatmulV5Test(int32_t deviceId, aclrtStream &stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Customize the construction based on the API.
    std::vector<int64_t> x1Shape = {16, 16};
    std::vector<int64_t> x2Shape = {16, 16};
    std::vector<int64_t> biasShape = {16};
    std::vector<int64_t> x2OffsetShape = {16};
    std::vector<int64_t> x1ScaleShape = {1};
    std::vector<int64_t> x2ScaleShape = {1};
    std::vector<int64_t> outShape = {16, 16};
    void *x1DeviceAddr = nullptr;
    void *x2DeviceAddr = nullptr;
    void *x2ScaleDeviceAddr = nullptr;
    void *x2OffsetDeviceAddr = nullptr;
    void *x1ScaleDeviceAddr = nullptr;
    void *biasDeviceAddr = nullptr;
    void *outDeviceAddr = nullptr;
    aclTensor *x1 = nullptr;
    aclTensor *x2 = nullptr;
    aclTensor *bias = nullptr;
    aclTensor *x2Scale = nullptr;
    aclTensor *x2Offset = nullptr;
    aclTensor *x1Scale = nullptr;
    aclTensor *out = nullptr;
    std::vector<int8_t> x1HostData(256, 1);
    std::vector<int8_t> x2HostData(256, 1);
    std::vector<int32_t> biasHostData(16, 1);
    std::vector<float> x2ScaleHostData(1, 1);
    std::vector<float> x2OffsetHostData(16, 1);
    std::vector<float> x1ScaleHostData(1, 1);
    std::vector<uint16_t> outHostData(256, 1);  // Actually in float16 half-precision format.

    // Create x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_FLOAT8_E4M3FN, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2 aclTensor.
    ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_FLOAT8_E4M3FN, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2Scale aclTensor.
    ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x2Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(x2Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x1Scale aclTensor.
    ret = CreateAclTensor(x1ScaleHostData, x1ScaleShape, &x1ScaleDeviceAddr,
                          aclDataType::ACL_FLOAT, &x1Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1ScaleTensorPtr(x1Scale,
                                                                                          aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1ScaleDeviceAddrPtr(x1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create bias aclTensor.
    ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = false;

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor *executor = nullptr;
    // Call the first-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5GetWorkspaceSize(x1, x2, x1Scale, x2Scale, nullptr, nullptr, x2Offset, nullptr, bias,
                                             transposeX1, transposeX2, 0, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5GetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void *workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Wait for the task execution to finish synchronously.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(
        size, 0);  // In C language, fp16 data cannot be printed directly. Read it as uint16 and convert it to fp16 manually through binary conversion.
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                      size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
    // Fill in the deviceId based on your actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulV5Test(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5Test failed. ERROR: %d\n", ret); return ret);

    Finalize(deviceId, stream);
    return 0;
}

• Atlas A2 training products/Atlas A2 inference products, Atlas A3 training products/Atlas A3 inference products:

x1 is INT8, x2 is INT32, x1Scale is FLOAT32, and x2Scale is UINT64.

#include <iostream>
#include <memory>
#include <vector>

#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul_v5.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define CHECK_FREE_RET(cond, return_expr) \
    do {                                  \
        if (!(cond)) {                    \
            Finalize(deviceId, stream);   \
            return_expr;                  \
        }                                 \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t> &shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to apply for device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Copy the host-side data to the device-side memory by calling aclrtMemcpy.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

    // Compute the strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call the aclCreateTensor API to create an aclTensor.
    *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
    return 0;
}

void Finalize(int32_t deviceId, aclrtStream stream)
{
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
}

int aclnnQuantMatmulV5Test(int32_t deviceId, aclrtStream &stream)
{
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Custom construction is required based on the API.
    std::vector<int64_t> x1Shape = {1, 8192};     // (m,k)
    std::vector<int64_t> x2Shape = {8192, 128};  // (k,n)
    std::vector<int64_t> yoffsetShape = {1024};

    std::vector<int64_t> x1ScaleShape = {1,1};
    std::vector<int64_t> x2ScaleShape = {32, 1024}; // x2ScaleShape = [KShape / groupsize, N]
    std::vector<int64_t> outShape = {1, 1024};

    void *x1DeviceAddr = nullptr;
    void *x2DeviceAddr = nullptr;
    void *x2ScaleDeviceAddr = nullptr;
    void *x1ScaleDeviceAddr = nullptr;
    void *yoffsetDeviceAddr = nullptr;
    void *outDeviceAddr = nullptr;
    aclTensor *x1 = nullptr;
    aclTensor *x2 = nullptr;
    aclTensor *yoffset = nullptr;
    aclTensor *x2Scale = nullptr;
    aclTensor *x2Offset = nullptr;
    aclTensor *x1Scale = nullptr;
    aclTensor *out = nullptr;
    std::vector<int8_t> x1HostData(GetShapeSize(x1Shape), 1);
    std::vector<int32_t> x2HostData(GetShapeSize(x2Shape), 1);
    std::vector<int32_t> yoffsetHostData(GetShapeSize(yoffsetShape), 1);
    std::vector<int32_t> x1ScaleHostData(GetShapeSize(x1ScaleShape), 1);
    float tmp = 1;
    uint64_t ans = static_cast<uint64_t>(*reinterpret_cast<int32_t*>(&tmp));
    std::vector<int64_t> x2ScaleHostData(GetShapeSize(x2ScaleShape), ans);
    std::vector<uint16_t> outHostData(GetShapeSize(outShape), 1);  // Actually in float16 half-precision format.

    // Create the x1 aclTensor.
    ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2 aclTensor.
    ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT32, &x2);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x1Scale aclTensor.
    ret = CreateAclTensor(x1ScaleHostData, x1ScaleShape, &x1ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x1Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1ScaleTensorPtr(x1Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x1ScaleDeviceAddrPtr(x1ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create x2Scale aclTensor.
    ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_UINT64, &x2Scale);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(x2Scale, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create yoffset aclTensor.
    ret = CreateAclTensor(yoffsetHostData, yoffsetShape, &yoffsetDeviceAddr, aclDataType::ACL_FLOAT, &yoffset);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> yoffsetTensorPtr(yoffset, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> yoffsetDeviceAddrPtr(yoffsetDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create out aclTensor.
    ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
    std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
    std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    bool transposeX1 = false;
    bool transposeX2 = false;
    int64_t groupSize = 256;

    // 3. Call the CANN operator library API. Replace it with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor *executor = nullptr;

    ret = aclnnQuantMatmulV5GetWorkspaceSize(x1, x2, x1Scale, x2Scale, nullptr, nullptr, nullptr, yoffset, nullptr,
                                            transposeX1, transposeX2, groupSize, out, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5GetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void *workspaceAddr = nullptr;
    std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
        workspaceAddrPtr.reset(workspaceAddr);
    }
    // Call the second-phase API of aclnnQuantMatmulV5.
    ret = aclnnQuantMatmulV5(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5 failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value. Copy the result from the device memory to the host memory. Modify based on the specific API definition.
    auto size = GetShapeSize(outShape);
    std::vector<uint16_t> resultData(size, 0);  // fp16 data cannot be directly printed in C. Use uint16 to read it, and then convert it to fp16 through binary conversion.
    ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
                    size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
            return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
    }
    return ACL_SUCCESS;
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 1;
    aclrtStream stream;
    auto ret = aclnnQuantMatmulV5Test(deviceId, stream);
    CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV5Test failed. ERROR: %d\n", ret); return ret);
    Finalize(deviceId, stream);
    return 0;
}