aclnnQuantMatmulV3
Notice: This API will be deprecated in later versions. Use the latest aclnnQuantMatmulV5 API instead.
Applicable Products
| Product | Supported |
|---|---|
| √ | |
| √ | |
| √ | |
| × | |
| √ | |
| × |
Function
Operator function: Performs quantized matrix multiplication. The minimum supported input dimension is 2, and the maximum supported input dimension is 6. Similar APIs include aclnnMm (matrix multiplication that only supports 2D tensors as input) and aclnnBatchMatMul (matrix multiplication that only supports 3D tensors, where the first dimension is the batch dimension). Supports T-C && T-T quant mode.
Formula:
Without bias:
bias INT32:
Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products ,Ascend 950PR/Ascend 950DT :Supports bias in BFLOAT16/FLOAT32 (no offset in this scenario).
Function Prototype
Each operator is divided into a two-phase API. The "aclnnQuantMatmulV3GetWorkspaceSize" API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the "aclnnQuantMatmulV3" API is called to perform the computation.
aclnnStatus aclnnQuantMatmulV3GetWorkspaceSize(
const aclTensor *x1,
const aclTensor *x2,
const aclTensor *scale,
const aclTensor *offset,
const aclTensor *bias,
bool transposeX1,
bool transposeX2,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnQuantMatmulV3(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnQuantMatmulV3GetWorkspaceSize
Parameters:
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor x1 Input Input x1 in the formula. - Supports non-contiguous tensors when the last two axes are transposed. Non-contiguous tensors in other scenarios are not supported.
- When transposeX1 is false, the shape is (batch, m, k). When transposeX1 is true, the shape is (batch, k, m). The batch dimension is optional.
INT8, INT32, INT4 ND 2-6 √ x2 Input Input x2 in the formula. - ND:
- Non-contiguous tensors with the last two axes transposed are supported. Non-contiguous tensors in other scenarios are not supported.
- When transposeX1 is false, the shape is (batch, n, k). When transposeX1 is true, the shape is (batch, k, n). batch may be absent, where k is consistent with k in the shape of x1.
- NZ:
- When transposeX2 is true, the shape is (batch, k1, n1, n0, k0). batch may be absent, where k0=32, n0=16, and k in the shape of x1 and k1 in the shape of x2 must satisfy ceil(k / 32) = k1.
- When transposeX2 is false, the shape is (batch, n1, k1, k0, n0). batch may be absent, where k0=16, n0=32, and k in the shape of x1 and k1 in the shape of x2 must satisfy ceil(k / 16) = k1.
INT8, INT32, INT4 ND, NZ 2-8 (ND), 4-8 (NZ) √ scale Input Represents the quantization parameter, i.e., the input scale in the formula. (t,), where t = 1 or n, and n is consistent with n of x2.
When the original input type does not meet the type combinations in [Constraints], call the aclnn API of the TransQuantParamV2 operator in advance to convert scale to INT64 or UINT64 type.UINT64, INT64, FLOAT32, BFLOAT16 ND 1 × offset Input Input offset in the formula. (t,), t = 1 or n, where n is consistent with n of x2.
When the out data type is INT8, offset can exist. For other input types, nullptr must be passed.FLOAT32 ND 1 × bias Optional input Input bias in the formula. The shape supports 1D (n,) or 3D (batch, 1, n), where n is consistent with n of x2.
When the shape of out is 2D, 4D, 5D, or 6D, the shape of bias only supports 1D.INT32, BFLOAT16, FLOAT32, ND 1, 3 × transposeX1 Input Indicates whether the input shape of x1 includes transpose. When transposeX1 is false, the shape is (batch, m, k).
When transposeX1 is true, the shape is (batch, k, m).
batch may be absent.bool - - × transposeX2 Input Indicates whether the input shape of x2 includes transpose. - ND: When transposeX2 is false, the shape is in the form of (batch, k, n).
When transposeX2 is true, the shape is in the form of (batch, n, k).
batch may be absent, where k is consistent with k in the shape of x1. - NZ:
- When transposeX2 is true, the shape is in the form of (batch, k1, n1, n0, k0). batch may be absent, where k0 = 32, n0 = 16, and k in the shape of x1 and k1 in the shape of x2 must satisfy the following relationship: ceil(k / 32) = k1.
- When transposeX2 is false, the shape is in the form of (batch, n1, k1, k0, n0). batch may be absent, where k0 = 16, n0 = 32, and k in the shape of x1 and k1 in the shape of x2 must satisfy the following relationship: ceil(k / 16) = k1.
bool - - × out Output The output "out" in the formula. (batch, m, n), where batch may be absent. Supports broadcasting of the batch dimensions of x1 and x2.
The output batch is consistent with the batch after broadcasting, m is consistent with m of x1, and n is consistent with n of x2.FLOAT16, INT8, BFLOAT16, INT32 ND 2-6 √ workspaceSize Output Returns the workspace size to be applied for on the Device side. - - - - - executor Output Returns the operator executor, which contains the operator computation flow. - - - - - Atlas inference products :x1 and x2 support INT8.
scale supports UINT64 and INT64.
bias supports INT32.
out supports FLOAT16 and INT8.
Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products :x1 and x2 support INT8, INT32, and INT4.
The scale data type supports UINT64, INT64, FLOAT32, and BFLOAT16.
bias supports INT32, BFLOAT16, and FLOAT32. When x1 and x2 are INT32 or INT4, the shape of bias only supports 1D (n,).
When x1 and x2 are INT32 or INT4, transposeX1 only supports false.
out supports FLOAT16, INT8, BFLOAT16, and INT32.
Ascend 950PR/Ascend 950DT :x1 and x2 support INT8.
The scale data type supports UINT64, INT64, FLOAT32, and BFLOAT16.
scale supports INT32, BFLOAT16, and FLOAT32.
out supports FLOAT16, INT8, BFLOAT16, and INT32.
When x2 is in ND format, if the input x1 is an empty tensor with m=0 or x2 is an empty tensor with n=0, the output is an empty tensor. When x2 is in FRACTAL_NZ format, if the input x1 is an empty tensor with m=0, the output is an empty tensor.
Return value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
| Return Value | Error Code | Description |
|---|---|---|
| ACLNN_ERR_PARAM_NULLPTR | 161001 | The passed x1, x2, scale, or out is a null pointer. |
| ACLNN_ERR_PARAM_INVALID | 161002 | The data type or data format of x1, x2, bias, scale, offset, or out is not within the supported range. |
| The shape of x1, x2, bias, scale, offset, or out does not meet the validation conditions. | ||
| x1, x2, bias, x2Scale, x2Offset, or out is an empty tensor. |
aclnnQuantMatmulV3
Parameters:
Parameters Input/Output Description workspace Input Workspace memory address applied for on the Device side. workspaceSize Input Workspace size applied for on the Device side, obtained through the first-phase API aclnnQuantMatmulV3GetWorkspaceSize. executor Input op executor, which contains the operator computation flow. stream Input Specifies the stream for task execution. Return value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic description:
Atlas training products andAtlas inference products : aclnnQuantMatmulV3 default to a deterministic implementation.Ascend 950PR/Ascend 950DT : aclnnQuantMatmulV3 defaults to a deterministic implementation.
Atlas inference products :The size of the last dimension of x1 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true, or k when transposeX1 is false.
The size of the last dimension of x2 cannot exceed 65535. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false.
- When the input x2 is in NZ format, the scenario where transposeX2 is false is not supported.
Before calling this API, you can process x2 in ND format to obtain the NZ format by calling aclnnTransMatmulWeight.
Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products :The size of the last dimension of x1 cannot exceed 65535. The last dimension of x1 refers to m when transposeX1 is true or k when transposeX1 is false. When the data type of x1 is INT32 or INT4, it is an INT4 quantization scenario, and currently only the case where transposeX1 is false is supported. When the data type of x1 is INT4, the dimension representation is (batch, m, k), and k must be an even number. When the data type of x1 is INT32, each INT32 value stores eight INT4 values, and the corresponding dimension representation is (batch, m, k // 8), and k must be a multiple of 8.
The size of the last dimension of x2 cannot exceed 65535. The last dimension of x2 refers to k when transposeX2 is true or n when transposeX2 is false. When the input x2 is in NZ format, the scenario where transposeX2 is false is not supported.
When the data type is INT4, if transposeX2 is true, the shape is (n, k) and k must be even; if transposeX2 is false, the shape is (k, n) and n must be even.
When the data type is INT32, each INT32 value stores eight INT4 values. If transposeX2 is true, the shape is (n, k // 8) and k must be a multiple of 8; if transposeX2 is false, the shape is (k, n // 8) and n must be a multiple of 8.
You can use the aclnnConvertWeightToINT4Pack API to convert x2 from INT32 (one INT32 stores one INT4 in bits 0–3) to INT32 (one INT32 stores eight INT4 values) or INT4 (one INT4 represents one INT4 value). For details, see the aclnnConvertWeightToINT4Pack API.
Before calling this API, you can process x2 in ND format to obtain the NZ format through aclnnTransMatmulWeight.
Ascend 950PR/Ascend 950DT :When one of the last two axes is 1 (i.e., n=1 or k=1), x2 does not support private formats and only supports the ND format.
Before calling this API, you can use aclnnTransMatmulWeight or aclnnNpuFormatCast to process x2 in ND format to obtain the NZ format.
When one of the last two dimensions of the original ND format is 1, conversion to NZ is not recommended. By default, x2 is non-contiguous, and only non-contiguous tensors are supported for x2.
The input and output support the following data type combinations, which support T-C && T-T quantization mode:
NOTE When the original input types do not meet the following type combinations, call the aclnn API of the TransQuantParamV2 operator in advance to convert scale to the INT64 or UINT64 data type.
Atlas inference products :x1 x2 scale offset bias out INT8 INT8 UINT64/INT64 null null/INT32 FLOAT16 INT8 INT8 UINT64/INT64 null/FLOAT32 null/INT32 INT8 Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products :x1 x2 scale offset bias out INT8 INT8 UINT64/INT64 null null/INT32 FLOAT16 INT8 INT8 UINT64/INT64 null/FLOAT32 null/INT32 INT8 INT8 INT8 FLOAT32/BFLOAT16 null null/INT32/BFLOAT16/FLOAT32 BFLOAT16 INT4/INT32 INT4/INT32 UINT64/INT64 null null/INT32 FLOAT16 INT8 INT8 FLOAT32/BFLOAT16 null null/INT32 INT32 Ascend 950PR/Ascend 950DT :x1 x2 scale offset bias out INT8 INT8 UINT64/INT64 null null/INT32 FLOAT16/BFLOAT16 INT8 INT8 UINT64/INT64 null/FLOAT32 null/INT32 INT8 INT8 INT8 FLOAT32/BFLOAT16 null null/INT32/BFLOAT16/FLOAT32 BFLOAT16 INT8 INT8 FLOAT32/BFLOAT16 null null/INT32 INT32
Example
Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products ,Ascend 950PR/Ascend 950DT :
The following provides example code for general scenarios for reference only. For details about compilation and execution, see Compile and Run Samples.
#include <iostream>
#include <memory>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul_v3.h"
#include "aclnnop/aclnn_trans_quant_param_v2.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define CHECK_FREE_RET(cond, return_expr) \
do { \
if (!(cond)) { \
Finalize(deviceId, stream); \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t> &shape)
{
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream *stream)
{
// Boilerplate: resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
aclDataType dataType, aclTensor **tensor)
{
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate device memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy host data to device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
void Finalize(int32_t deviceId, aclrtStream stream)
{
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
}
int aclnnQuantMatmulV3Test(int32_t deviceId, aclrtStream &stream)
{
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct inputs and outputs. Customize the construction based on the API.
std::vector<int64_t> x1Shape = {5, 2};
std::vector<int64_t> x2Shape = {2, 3};
std::vector<int64_t> biasShape = {3};
std::vector<int64_t> offsetShape = {3};
std::vector<int64_t> scaleShape = {3};
std::vector<int64_t> outShape = {5, 3};
void *x1DeviceAddr = nullptr;
void *x2DeviceAddr = nullptr;
void *scaleDeviceAddr = nullptr;
void *quantParamDeviceAddr = nullptr;
void *offsetDeviceAddr = nullptr;
void *biasDeviceAddr = nullptr;
void *outDeviceAddr = nullptr;
aclTensor *x1 = nullptr;
aclTensor *x2 = nullptr;
aclTensor *bias = nullptr;
aclTensor *scale = nullptr;
aclTensor *quantParam = nullptr;
aclTensor *offset = nullptr;
aclTensor *out = nullptr;
std::vector<int8_t> x1HostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
std::vector<int8_t> x2HostData = {1, 1, 1, 1, 1, 1};
std::vector<int32_t> biasHostData = {1, 1, 1};
std::vector<float> scaleHostData = {1, 1, 1};
std::vector<float> offsetHostData = {1, 1, 1};
std::vector<uint16_t> outHostData = {1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1}; // Actually in float16 half-precision format.
// Create the x1 aclTensor.
ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x2 aclTensor.
ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create scale aclTensor.
ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create quantParam aclTensor.
ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam,
aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create offset aclTensor.
ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> offsetTensorPtr(offset, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> offsetDeviceAddrPtr(offsetDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create bias aclTensor.
ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create out aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
bool transposeX1 = false;
bool transposeX2 = false;
// 3. Call the CANN operator library API. Replace with the specific API name.
uint64_t workspaceSize = 0;
aclOpExecutor *executor = nullptr;
// For FLOAT data type scale, call the aclnnTransQuantParamV2 operator's aclnn API in advance.
// Call the first-phase API of aclnnTransQuantParamV2.
ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret);
return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
void *workspaceAddr = nullptr;
std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree);
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
workspaceAddrPtrV2.reset(workspaceAddr);
}
// Call the second-phase API of aclnnTransQuantParamV2.
ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret);
// Call the first-phase API of aclnnQuantMatmulV3.
ret = aclnnQuantMatmulV3GetWorkspaceSize(x1, x2, quantParam, nullptr, bias, transposeX1, transposeX2, out,
&workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3GetWorkspaceSize failed. ERROR: %d\n", ret);
return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
workspaceAddr = nullptr;
std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV3(nullptr, aclrtFree);
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
workspaceAddrPtrV3.reset(workspaceAddr);
}
// Call the second-phase API of aclnnQuantMatmulV3.
ret = aclnnQuantMatmulV3(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3 failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronize and wait for task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
auto size = GetShapeSize(outShape);
std::vector<uint16_t> resultData(
size, 0); // fp16 data cannot be directly printed in C. Read it as uint16 and manually convert it to fp16 via binary.
ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
return ret);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("result[%ld] is: %hu\n", i, resultData[i]);
}
return ACL_SUCCESS;
}
int main()
{
// 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
// Set deviceId based on your actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = aclnnQuantMatmulV3Test(deviceId, stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3Test failed. ERROR: %d\n", ret); return ret);
Finalize(deviceId, stream);
return 0;
}Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products : The following is a sample code for the scenario where x2 is in NZ format (transposeX2=false), for reference only. For details about compilation and execution, see Compile and Run Samples.#include <iostream> #include <memory> #include <vector> #include "acl/acl.h" #include "aclnnop/aclnn_permute.h" #include "aclnnop/aclnn_quant_matmul_v3.h" #include "aclnnop/aclnn_trans_matmul_weight.h" #include "aclnnop/aclnn_trans_quant_param_v2.h" #define CHECK_RET(cond, return_expr) \ do { \ if (!(cond)) { \ return_expr; \ } \ } while (0) #define CHECK_FREE_RET(cond, return_expr) \ do { \ if (!(cond)) { \ Finalize(deviceId, stream); \ return_expr; \ } \ } while (0) #define LOG_PRINT(message, ...) \ do { \ printf(message, ##__VA_ARGS__); \ } while (0) int64_t GetShapeSize(const std::vector<int64_t> &shape) { int64_t shapeSize = 1; for (auto i : shape) { shapeSize *= i; } return shapeSize; } int Init(int32_t deviceId, aclrtStream *stream) { // Boilerplate: resource initialization. auto ret = aclInit(nullptr); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret); ret = aclrtSetDevice(deviceId); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret); ret = aclrtCreateStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret); return 0; } template <typename T> int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { auto size = GetShapeSize(shape) * sizeof(T); // Call aclrtMalloc to apply for device-side memory. auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret); // Call aclrtMemcpy to copy host-side data to device-side memory. ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret); // Calculate the strides of a contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } // Call the aclCreateTensor API to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(), *deviceAddr); return 0; } void Finalize(int32_t deviceId, aclrtStream stream) { aclrtDestroyStream(stream); aclrtResetDevice(deviceId); aclFinalize(); } template <typename T> int CreateAclTensorX2(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { auto size = static_cast<uint64_t>(GetShapeSize(shape)); const aclIntArray *mat2Size = aclCreateIntArray(shape.data(), shape.size()); auto ret = aclnnCalculateMatmulWeightSizeV2(mat2Size, dataType, &size); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSizeV2 failed. ERROR: %d\n", ret); return ret); size *= sizeof(T); // Call aclrtMalloc to apply for device-side memory. ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret); // Copy data from the host side to the device memory. ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret); // Calculate the strides of a contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } std::vector<int64_t> storageShape; storageShape.push_back(GetShapeSize(shape)); // Call the aclCreateTensor API to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, storageShape.data(), storageShape.size(), *deviceAddr); return 0; } int aclnnQuantMatmulV3Test(int32_t deviceId, aclrtStream &stream) { auto ret = Init(deviceId, &stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret); // 2. Construct the inputs and outputs. Custom construction is required based on the API. std::vector<int64_t> x1Shape = {5, 2}; std::vector<int64_t> x2Shape = {2, 3}; std::vector<int64_t> biasShape = {3}; std::vector<int64_t> offsetShape = {3}; std::vector<int64_t> scaleShape = {3}; std::vector<int64_t> outShape = {5, 3}; void *x1DeviceAddr = nullptr; void *x2DeviceAddr = nullptr; void *scaleDeviceAddr = nullptr; void *quantParamDeviceAddr = nullptr; void *offsetDeviceAddr = nullptr; void *biasDeviceAddr = nullptr; void *outDeviceAddr = nullptr; aclTensor *x1 = nullptr; aclTensor *x2 = nullptr; aclTensor *bias = nullptr; aclTensor *scale = nullptr; aclTensor *quantParam = nullptr; aclTensor *offset = nullptr; aclTensor *out = nullptr; std::vector<int8_t> x1HostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1}; std::vector<int8_t> x2HostData = {1, 1, 1, 1, 1, 1}; std::vector<int32_t> biasHostData = {1, 1, 1}; std::vector<float> scaleHostData = {1, 1, 1}; std::vector<float> offsetHostData = {1, 1, 1}; std::vector<uint16_t> outHostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}; // Actually in float16 half-precision mode. // Create the x1 aclTensor. ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the x2 aclTensor in NZ format. ret = CreateAclTensorX2(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2HPTensorPtr(x2, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x2HPDeviceAddrPtr(x2DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the scale aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the quantParam aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the offset aclTensor. ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> offsetTensorPtr(offset, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> offsetDeviceAddrPtr(offsetDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the bias aclTensor. ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the out aclTensor. ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); bool transposeX1 = false; bool transposeX2 = false; // 3. Call the CANN operator library API. Replace with the specific API name. uint64_t workspaceSize = 0; aclOpExecutor *executor = nullptr; void *workspaceAddr = nullptr; // Call the first-phase API of aclnnTransMatmulWeight. ret = aclnnTransMatmulWeightGetWorkspaceSize(x2, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeightGetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrTrans(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrTrans.reset(workspaceAddr); } // Call the second-phase API of aclnnTransMatmulWeight. ret = aclnnTransMatmulWeight(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeight failed. ERROR: %d\n", ret); return ret); // For the scale factor of FLOAT data type, call the aclnn API of the TransQuantParamV2 operator in advance. // Call the first-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. workspaceAddr = nullptr; std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV2.reset(workspaceAddr); } // Call the second-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret); // Call the first-phase API of aclnnQuantMatmulV3. workspaceSize = 0; ret = aclnnQuantMatmulV3GetWorkspaceSize(x1, x2, quantParam, nullptr, bias, transposeX1, transposeX2, out, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV3(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV3.reset(workspaceAddr); } // Call the second-phase API of aclnnQuantMatmulV3. ret = aclnnQuantMatmulV3(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3 failed. ERROR: %d\n", ret); return ret); // 4. (Boilerplate) Synchronize and wait for task execution to complete. ret = aclrtSynchronizeStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret); // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition. auto size = GetShapeSize(outShape); std::vector<uint16_t> resultData( size, 0); // fp16 data cannot be directly printed in C. Read it as uint16 and manually convert it to fp16 via binary. ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret); for (int64_t i = 0; i < size; i++) { LOG_PRINT("result[%ld] is: %hu\n", i, resultData[i]); } return ACL_SUCCESS; } int main() { // 1. (Boilerplate) Initialize the device/stream. Refer to the acl API manual. // Set deviceId based on your actual device. int32_t deviceId = 0; aclrtStream stream; auto ret = aclnnQuantMatmulV3Test(deviceId, stream); CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3Test failed. ERROR: %d\n", ret); return ret); Finalize(deviceId, stream); return 0; }Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products : The following is sample code for the INT4 quantization scenario (x1 and x2 data type is INT4, transposeX2=false), for reference only. For the specific compilation and execution process, see Compile and Run Samples.#include <iostream> #include <memory> #include <vector> #include "acl/acl.h" #include "aclnnop/aclnn_convert_weight_to_int4_pack.h" #include "aclnnop/aclnn_quant_matmul_v3.h" #include "aclnnop/aclnn_trans_quant_param_v2.h" #define CHECK_RET(cond, return_expr) \ do { \ if (!(cond)) { \ return_expr; \ } \ } while (0) #define CHECK_FREE_RET(cond, return_expr) \ do { \ if (!(cond)) { \ Finalize(deviceId, stream); \ return_expr; \ } \ } while (0) #define LOG_PRINT(message, ...) \ do { \ printf(message, ##__VA_ARGS__); \ } while (0) int64_t GetShapeSize(const std::vector<int64_t> &shape) { int64_t shapeSize = 1; for (auto i : shape) { shapeSize *= i; } return shapeSize; } int Init(int32_t deviceId, aclrtStream *stream) { // Boilerplate: resource initialization. auto ret = aclInit(nullptr); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret); ret = aclrtSetDevice(deviceId); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret); ret = aclrtCreateStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret); return 0; } template <typename T> int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { // Obtain the number of bytes for allocation and copy from hostData. auto size = hostData.size() * sizeof(T); // Call aclrtMalloc to allocate device-side memory. auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret); // Call aclrtMemcpy to copy host-side data to device-side memory. ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret); // Compute the strides of a contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } // Call the aclCreateTensor API to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(), *deviceAddr); return 0; } void Finalize(int32_t deviceId, aclrtStream stream) { aclrtDestroyStream(stream); aclrtResetDevice(deviceId); aclFinalize(); } int aclnnQuantMatmulV3Test(int32_t deviceId, aclrtStream &stream) { auto ret = Init(deviceId, &stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret); // 2. Construct inputs and outputs. Custom construction is required based on the API. int64_t m = 16; int64_t k = 8; int64_t n = 32; aclDataType x1Dtype = aclDataType::ACL_INT4; aclDataType x2Int4PackDtype = aclDataType::ACL_INT4; std::vector<int64_t> x1Shape = {m, k}; std::vector<int64_t> x2Shape = {k, n}; std::vector<int64_t> x2Int4PackShape = {k, n}; std::vector<int64_t> biasShape = {n}; std::vector<int64_t> scaleShape = {n}; std::vector<int64_t> outShape = {m, n}; void *x1DeviceAddr = nullptr; void *x2DeviceAddr = nullptr; void *x2Int4PackDeviceAddr = nullptr; void *scaleDeviceAddr = nullptr; void *quantParamDeviceAddr = nullptr; void *offsetDeviceAddr = nullptr; void *biasDeviceAddr = nullptr; void *outDeviceAddr = nullptr; aclTensor *x1 = nullptr; aclTensor *x2 = nullptr; aclTensor *x2Int4Pack = nullptr; aclTensor *bias = nullptr; aclTensor *scale = nullptr; aclTensor *quantParam = nullptr; aclTensor *offset = nullptr; aclTensor *out = nullptr; std::vector<int8_t> x1HostData(m * k / 2, 17); // int8: 0001 0001 std::vector<int8_t> x2HostData(k * n, 1); std::vector<int8_t> x2Int4PackHostData(n * k / 2, 1); std::vector<int32_t> biasHostData(n, 1); std::vector<float> scaleHostData(n, 1); std::vector<uint16_t> outHostData(m * n, 1); // Create x1 aclTensor. ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, x1Dtype, &x1); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create x2 aclTensor. ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT32, &x2); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create x2Int4Pack aclTensor. ret = CreateAclTensor(x2Int4PackHostData, x2Int4PackShape, &x2Int4PackDeviceAddr, x2Int4PackDtype, &x2Int4Pack); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2Int4PackTensorPtr(x2Int4Pack, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x2Int4PackDeviceAddrPtr(x2Int4PackDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create scale aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the quantParam aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the bias aclTensor. ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the out aclTensor. ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); bool transposeX1 = false; bool transposeX2 = false; // 3. Call the CANN operator library API. Replace with the specific API name. uint64_t workspaceSize = 0; aclOpExecutor *executor = nullptr; // You can first call the aclnnConvertWeightToINT4Pack API to construct the x2 input data. // Call the first-phase API of aclnnConvertWeightToINT4Pack. ret = aclnnConvertWeightToINT4PackGetWorkspaceSize(x2, x2Int4Pack, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnConvertWeightToINT4PackGetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. void *workspaceAddr = nullptr; std::unique_ptr<void, aclError (*)(void *)> workspaceINT4PackAddrPtr(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceINT4PackAddrPtr.reset(workspaceAddr); } // Call the second-phase API of aclnnConvertWeightToINT4Pack. ret = aclnnConvertWeightToINT4Pack(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnConvertWeightToINT4Pack failed. ERROR: %d\n", ret); return ret); // For the scale of the FLOAT data type, call the aclnn API of the TransQuantParamV2 operator in advance. // Call the first-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. workspaceAddr = nullptr; std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV2.reset(workspaceAddr); } // Call the second-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret); // Call the first-phase API of aclnnQuantMatmulV3. ret = aclnnQuantMatmulV3GetWorkspaceSize(x1, x2Int4Pack, quantParam, nullptr, bias, transposeX1, transposeX2, out, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. workspaceAddr = nullptr; std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV3(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV3.reset(workspaceAddr); } // Call the second-phase API of aclnnQuantMatmulV3. ret = aclnnQuantMatmulV3(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3 failed. ERROR: %d\n", ret); return ret); // 4. (Boilerplate) Synchronously wait for task execution to finish. ret = aclrtSynchronizeStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret); // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition. auto size = GetShapeSize(outShape); std::vector<uint16_t> resultData( size, 0); // In C, fp16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 via binary representation. ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret); for (int64_t i = 0; i < size; i++) { LOG_PRINT("result[%ld] is: %hu\n", i, resultData[i]); } return ACL_SUCCESS; } int main() { // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual. // Enter the actual device ID for deviceId. int32_t deviceId = 0; aclrtStream stream; auto ret = aclnnQuantMatmulV3Test(deviceId, stream); CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3Test failed. ERROR: %d\n", ret); return ret); Finalize(deviceId, stream); return 0; }Atlas inference products : The sample code for the scenario where x2 is in NZ format (transposeX2=true) is as follows, for reference only. For the specific compilation and execution process, see Compile and Run Samples.#include <iostream> #include <memory> #include <vector> #include "acl/acl.h" #include "aclnnop/aclnn_permute.h" #include "aclnnop/aclnn_quant_matmul_v3.h" #include "aclnnop/aclnn_trans_matmul_weight.h" #include "aclnnop/aclnn_trans_quant_param_v2.h" #define CHECK_RET(cond, return_expr) \ do { \ if (!(cond)) { \ return_expr; \ } \ } while (0) #define CHECK_FREE_RET(cond, return_expr) \ do { \ if (!(cond)) { \ Finalize(deviceId, stream); \ return_expr; \ } \ } while (0) #define LOG_PRINT(message, ...) \ do { \ printf(message, ##__VA_ARGS__); \ } while (0) int64_t GetShapeSize(const std::vector<int64_t> &shape) { int64_t shapeSize = 1; for (auto i : shape) { shapeSize *= i; } return shapeSize; } int Init(int32_t deviceId, aclrtStream *stream) { // Boilerplate: resource initialization. auto ret = aclInit(nullptr); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret); ret = aclrtSetDevice(deviceId); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret); ret = aclrtCreateStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret); return 0; } template <typename T> int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { auto size = GetShapeSize(shape) * sizeof(T); // Call aclrtMalloc to apply for device-side memory. auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret); // Call aclrtMemcpy to copy host-side data to the device-side memory. ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret); // Compute the strides of a contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } // Call aclCreateTensor to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(), *deviceAddr); return 0; } void Finalize(int32_t deviceId, aclrtStream stream) { aclrtDestroyStream(stream); aclrtResetDevice(deviceId); aclFinalize(); } template <typename T> int CreateAclTensorX2(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr, aclDataType dataType, aclTensor **tensor) { auto size = static_cast<uint64_t>(GetShapeSize(shape)); const aclIntArray *mat2Size = aclCreateIntArray(shape.data(), shape.size()); auto ret = aclnnCalculateMatmulWeightSizeV2(mat2Size, dataType, &size); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSizeV2 failed. ERROR: %d\n", ret); return ret); size *= sizeof(T); // Call aclrtMalloc to apply for device-side memory. ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret); // Call aclrtMemcpy to copy data from the host side to the device-side memory. ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret); // Calculate the strides of a contiguous tensor. std::vector<int64_t> strides(shape.size(), 1); for (int64_t i = shape.size() - 2; i >= 0; i--) { strides[i] = shape[i + 1] * strides[i + 1]; } std::vector<int64_t> storageShape; storageShape.push_back(GetShapeSize(shape)); // Call aclCreateTensor to create an aclTensor. *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, storageShape.data(), storageShape.size(), *deviceAddr); return 0; } int aclnnQuantMatmulV3Test(int32_t deviceId, aclrtStream &stream) { auto ret = Init(deviceId, &stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret); // 2. Construct the input and output. Customize the construction based on the API. std::vector<int64_t> x1Shape = {5, 2}; std::vector<int64_t> x2Shape = {2, 3}; std::vector<int64_t> x2TransposedShape = {3, 2}; std::vector<int64_t> biasShape = {3}; std::vector<int64_t> offsetShape = {3}; std::vector<int64_t> scaleShape = {3}; std::vector<int64_t> outShape = {5, 3}; void *x1DeviceAddr = nullptr; void *x2DeviceAddr = nullptr; void *x2TransposedDeviceAddr = nullptr; void *scaleDeviceAddr = nullptr; void *quantParamDeviceAddr = nullptr; void *offsetDeviceAddr = nullptr; void *biasDeviceAddr = nullptr; void *outDeviceAddr = nullptr; aclTensor *x1 = nullptr; aclTensor *x2 = nullptr; aclTensor *x2Transposed = nullptr; aclTensor *bias = nullptr; aclTensor *scale = nullptr; aclTensor *quantParam = nullptr; aclTensor *offset = nullptr; aclTensor *out = nullptr; std::vector<int8_t> x1HostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1}; std::vector<int8_t> x2HostData = {1, 1, 1, 1, 1, 1}; std::vector<int8_t> x2TransposedHostData = {1, 1, 1, 1, 1, 1}; std::vector<int32_t> biasHostData = {1, 1, 1}; std::vector<float> scaleHostData = {1, 1, 1}; std::vector<float> offsetHostData = {1, 1, 1}; std::vector<uint16_t> outHostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1}; // Actually in float16 half-precision format. // Create the x1 aclTensor. ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the x2 aclTensor in NZ format. ret = CreateAclTensorX2(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2HPTensorPtr(x2, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x2HPDeviceAddrPtr(x2DeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the x2Transposed aclTensor in NZ format. ret = CreateAclTensorX2(x2TransposedHostData, x2TransposedShape, &x2TransposedDeviceAddr, aclDataType::ACL_INT8, &x2Transposed); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TransposedHPTensorPtr(x2Transposed, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> x2TransposedHPDeviceAddrPtr(x2TransposedDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the scale aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &scaleDeviceAddr, aclDataType::ACL_FLOAT, &scale); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> scaleTensorPtr(scale, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> scaleDeviceAddrPtr(scaleDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the quantParam aclTensor. ret = CreateAclTensor(scaleHostData, scaleShape, &quantParamDeviceAddr, aclDataType::ACL_UINT64, &quantParam); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> quantParamTensorPtr(quantParam, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> quantParamDeviceAddrPtr(quantParamDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the offset aclTensor. ret = CreateAclTensor(offsetHostData, offsetShape, &offsetDeviceAddr, aclDataType::ACL_FLOAT, &offset); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> offsetTensorPtr(offset, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> offsetDeviceAddrPtr(offsetDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the bias aclTensor. ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); // Create the out aclTensor. ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out); std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor); std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree); CHECK_RET(ret == ACL_SUCCESS, return ret); bool transposeX1 = false; bool transposeX2 = true; // 3. Call the CANN operator library API. Replace with the specific API name. uint64_t workspaceSize = 0; aclOpExecutor *executor = nullptr; void *workspaceAddr = nullptr; // Transpose the shape of x2 to nk format, then perform transdata. std::vector<int64_t> dimsData = {1, 0}; // Create dims aclIntArray. aclIntArray *dims = aclCreateIntArray(dimsData.data(), dimsData.size()); // Call the first-phase API of aclnnPermute. ret = aclnnPermuteGetWorkspaceSize(x2, dims, x2Transposed, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnPermuteGetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Allocate device memory based on workspaceSize calculated by the first-phase API. std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrPermute(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrPermute.reset(workspaceAddr); } // Call the second-phase API of aclnnPermute. ret = aclnnPermute(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnPermuteGetWorkspaceSize failed. ERROR: %d\n", ret); return ret); workspaceSize = 0; // Call the first-phase API of aclnnTransMatmulWeight. ret = aclnnTransMatmulWeightGetWorkspaceSize(x2Transposed, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeightGetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrTrans(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrTrans.reset(workspaceAddr); } // Call the second-phase API of aclnnTransMatmulWeight. ret = aclnnTransMatmulWeight(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeight failed. ERROR: %d\n", ret); return ret); // The scale of the FLOAT data type needs to call the aclnn API of the TransQuantParamV2 operator in advance. // Call the first-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2GetWorkspaceSize(scale, offset, quantParam, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. workspaceAddr = nullptr; std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV2(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV2.reset(workspaceAddr); } // Call the second-phase API of aclnnTransQuantParamV2. ret = aclnnTransQuantParamV2(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransQuantParamV2 failed. ERROR: %d\n", ret); return ret); // Call the first-phase API of aclnnQuantMatmulV3. workspaceSize = 0; ret = aclnnQuantMatmulV3GetWorkspaceSize(x1, x2Transposed, quantParam, nullptr, bias, transposeX1, transposeX2, out, &workspaceSize, &executor); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3GetWorkspaceSize failed. ERROR: %d\n", ret); return ret); // Apply for device memory based on the workspaceSize calculated by the first-phase API. std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtrV3(nullptr, aclrtFree); if (workspaceSize > 0) { ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret); workspaceAddrPtrV3.reset(workspaceAddr); } // Call the second-phase API of aclnnQuantMatmulV3. ret = aclnnQuantMatmulV3(workspaceAddr, workspaceSize, executor, stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3 failed. ERROR: %d\n", ret); return ret); // 4. (Boilerplate) Wait for the task execution to finish synchronously. ret = aclrtSynchronizeStream(stream); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret); // 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition. auto size = GetShapeSize(outShape); std::vector<uint16_t> resultData( size, 0); // In C, fp16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 via binary representation. ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST); CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret); for (int64_t i = 0; i < size; i++) { LOG_PRINT("result[%ld] is: %hu\n", i, resultData[i]); } return ACL_SUCCESS; } int main() { // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual. // Fill in the actual device ID. // Fill in the deviceId based on your actual device. int32_t deviceId = 0; aclrtStream stream; auto ret = aclnnQuantMatmulV3Test(deviceId, stream); CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulV3Test failed. ERROR: %d\n", ret); return ret); Finalize(deviceId, stream); return 0; }