aclnnWeightQuantBatchMatmul
Note: This API will be deprecated in later versions. Use the latest aclnnWeightQuantBatchMatmulV3 API instead.
Applicable Products
| Product | Supported |
|---|---|
| × | |
| √ | |
| √ | |
| × | |
| × | |
| × |
Function
Description: Pseudo-quantization is used to quantize mat2 in self * mat2 (matmul/batchmatmul).
Formula:
Function Prototype
- Two-phase API. You must first call the aclnnWeightQuantBatchMatmulGetWorkspaceSize API to obtain the input parameters and calculate the required workspace size based on the computation process, and then call the aclnnWeightQuantBatchMatmul API to perform the computation.
aclnnStatus aclnnWeightQuantBatchMatmulGetWorkspaceSize(
const aclTensor *x1,
const aclTensor *x2,
const aclTensor *diagonalMatrix,
const aclTensor *deqOffset,
const aclTensor *deqScale,
const aclTensor *addOffset,
const aclTensor *mulScale,
const aclTensor *bias,
bool transposeX1,
bool transposeX2,
float antiquantScale,
float antiquantOffset,
aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnWeightQuantBatchMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
const aclrtStream stream)aclnnWeightQuantBatchMatmulGetWorkspaceSize
Parameters
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor x1 Input The input self in the formula. The dimension supports only 2D without the batch axis, and must satisfy the broadcast relationship with x2. FLOAT16 ND 2 - x2 Input After processing, it can be used to obtain the input mat2 in the formula. Only 2D is supported (batch axis is not supported), but it must satisfy the broadcast relationship with x1. INT8 ND 2 - diagonalMatrix Input Dequantizes x2 to obtain the input mat2 in the formula. Shape (32, 32), an identity matrix. Does not participate in calculation and can be considered empty when m > 64. INT8 ND 2D, with a shape of (32, 32). - deqOffset Input Input mat2 in the formula, dequantized from x2 and calculated from addOffset, antiQuantOffset, and antiQuantScale. For the calculation method, see the sample code. The shape supports 1, n, (1, 1), (1, n), or (n, 1), and must satisfy the broadcast relationship with x2. When m > 64, it does not participate in the calculation and can be considered empty. INT32 ND n, (1, 1), (1, n), or (n, 1) - deqScale Input Input mat2 in the formula, dequantized from x2 and calculated by the aclnnTransQuantParam API. For the calculation method, see the sample code. The shape supports 1, n, (1, 1), (1, n), or (n, 1), and must satisfy the broadcast relationship with x2. When m > 64, it does not participate in the calculation and can be considered empty. UINT64 ND 1 or n or (1, 1) or (1, n) or (n, 1) - addOffset Input Dequantizes x2 to obtain the input mat2 in the formula. The shape supports 1, n, (1, 1), (1, n), or (n, 1), and must satisfy the broadcast relationship with x2. When m < 64, it does not participate in the calculation and can be considered empty in any case. FLOAT16 ND 1 or n or (1, 1) or (1, n) or (n, 1) - mulScale Input Dequantizes x2 to obtain the input mat2 in the formula. The shape supports 1, n, (1, 1), (1, n), or (n, 1), and must be consistent with x2 to satisfy the broadcast relationship. It does not participate when m < 64 and can be considered empty in any case. FLOAT16 ND 1, n, (1, 1), (1, n), or (n, 1) - bias Input Input bias in the formula. Has one dimension with a value equal to N, and can be considered empty. FLOAT ND 1 - transposeX1 Input. Indicates whether x1 is transposed. - bool - - - transposeX2 Input Describes whether x2 is transposed. - bool - - - antiquantScale Input x2 is dequantized to obtain the input mat2 in the formula. - float - - - antiquantOffset Input Dequantizes x2 to obtain the input mat2 in the formula. - float - - - out Output The result in the formula. The data type must be a convertible data type derived from x1 and x2, and the shape must be the broadcast shape of x1 and x2. FLOAT16,INT8 ND - - workspaceSize Output Returns the workspace size to be applied on the Device side. - - - - - executor Output Returns the operator executor, which contains the operator computation flow. - - - - - Return value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 x1 or x2, diagonalMatrix (when m < 64), deqOffset (when m < 64), or deqScale (when m < 64) passed in is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type of the non-empty tensor passed in is not within the supported range.
aclnnWeightQuantBatchMatmul
Parameters
Parameter Input/Output Description workspace Input Memory address of the workspace applied on the Device side. workspaceSize Input Size of the workspace applied on the Device side, obtained from the first-phase API aclnnWeightQuantBatchMatmulGetWorkspaceSize. executor Input Operator executor that contains the operator computation process. stream Input Specifies the stream for executing the task. Return value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic description:
Atlas A3 training products/Atlas A3 inference products ,Atlas A2 training products/Atlas A2 inference products : aclnnWeightQuantBatchMatmul defaults to a deterministic implementation.
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include <cmath>
#include "acl/acl.h"
#include "aclnnop/aclnn_trans_quant_param.h"
#include "aclnnop/aclnn_cast.h"
#include "aclnnop/aclnn_weight_quant_batch_matmul.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate for resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to apply for device-side memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy host-side data to device-side memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
int main() {
// 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
// Fill in the deviceId based on your actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. Custom construction is required based on the API.
std::vector<int64_t> x1Shape = {128, 4};
std::vector<int64_t> x2Shape = {4, 4};
std::vector<int64_t> addOffsetShape = {4};
std::vector<int64_t> mulScaleShape = {4};
std::vector<int64_t> diagonalMatrixShape = {32, 32};
std::vector<int64_t> deqOffsetShape = {4};
std::vector<int64_t> deqScaleShape = {4};
std::vector<int64_t> outShape = {128, 4};
void* x1DeviceAddr = nullptr;
void* x2DeviceAddr = nullptr;
void* addOffsetDeviceAddr = nullptr;
void* mulScaleDeviceAddr = nullptr;
void* diagonalMatrixDeviceAddr = nullptr;
void* deqOffsetDeviceAddr = nullptr;
void* deqScaleDeviceAddr = nullptr;
void* outDeviceAddr = nullptr;
void* x1Fp16DeviceAddr = nullptr;
void* addOffsetFp16DeviceAddr = nullptr;
void* mulScaleFp16DeviceAddr = nullptr;
void* outFp16DeviceAddr = nullptr;
std::vector<float> x1HostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
std::vector<float> x2HostData = {1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1,
1, 1, 1, 1};
std::vector<float> outHostData = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
bool transposeX1 = false;
bool transposeX2 = false;
float antiquantOffset = 0;
float antiquantScale = 1;
std::vector<float> addOffsetHostData = {1.0, 1.0, 1.0, 0.0};
float* addOffsetData = addOffsetHostData.data();
uint64_t addOffsetSize = 4;
std::vector<float> mulScaleHostData = {2.0, 2.0, 1.0, 1.0};
float* mulScaleData = mulScaleHostData.data();
uint64_t mulScaleSize = 4;
// diagonalMatrixData
uint64_t n = 32;
uint64_t diagonalMatrixSize = n*n;
int8_t *diagonalMatrixData = (int8_t *)calloc(diagonalMatrixSize, sizeof(int32_t));
for (int64_t i = 0; i < n; i++) {
diagonalMatrixData[i * n + i] = 1;
}
std::vector<int8_t> diagonalMatrixHostData(diagonalMatrixData, diagonalMatrixData + diagonalMatrixSize);
// Get deqOffset
uint64_t deqOffsetSize = addOffsetSize;
int32_t *deqOffsetData = (int32_t *)calloc(deqOffsetSize, sizeof(int32_t));
for (int64_t i = 0; i < deqOffsetSize; i++) {
deqOffsetData[i] = static_cast<int32_t>(round(addOffsetData[i] / antiquantScale - antiquantOffset));
}
std::vector<int32_t> deqOffsetHostData(deqOffsetData, deqOffsetData + deqOffsetSize);
// Get deqScale
uint64_t deqScaleSize = mulScaleSize;
uint64_t *deqScaleData = (uint64_t *)calloc(deqScaleSize, sizeof(uint64_t));
for (int64_t i = 0; i < deqScaleSize; i++) {
mulScaleData[i] = mulScaleData[i] * antiquantScale;
}
std::vector<uint64_t> deqScaleHostData(deqScaleData, deqScaleData + deqScaleSize);
// creat aclTensor
aclTensor* x1 = nullptr;
aclTensor* x2 = nullptr;
aclTensor* addOffset = nullptr;
aclTensor* mulScale = nullptr;
aclTensor* diagonalMatrix = nullptr;
aclTensor* deqOffset = nullptr;
aclTensor* deqScale = nullptr;
aclTensor* out = nullptr;
aclTensor* x1Fp16 = nullptr;
aclTensor* addOffsetFp16 = nullptr;
aclTensor* mulScaleFp16 = nullptr;
aclTensor* outFp16 = nullptr;
// Create the x1 aclTensor.
ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_FLOAT, &x1);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x1Fp16 aclTensor.
ret = CreateAclTensor(x1HostData, x1Shape, &x1Fp16DeviceAddr, aclDataType::ACL_FLOAT16, &x1Fp16);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x2 aclTensor.
ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the addOffset aclTensor.
ret = CreateAclTensor(addOffsetHostData, addOffsetShape, &addOffsetDeviceAddr, aclDataType::ACL_FLOAT, &addOffset);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the addOffsetFp16 aclTensor.
ret = CreateAclTensor(addOffsetHostData, addOffsetShape, &addOffsetFp16DeviceAddr, aclDataType::ACL_FLOAT16, &addOffsetFp16);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create mulScale aclTensor.
ret = CreateAclTensor(mulScaleHostData, mulScaleShape, &mulScaleDeviceAddr, aclDataType::ACL_FLOAT, &mulScale);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create mulScaleFp16 aclTensor.
ret = CreateAclTensor(mulScaleHostData, mulScaleShape, &mulScaleFp16DeviceAddr, aclDataType::ACL_FLOAT16, &mulScaleFp16);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create diagonalMatrix aclTensor.
ret = CreateAclTensor(diagonalMatrixHostData, diagonalMatrixShape, &diagonalMatrixDeviceAddr, aclDataType::ACL_INT8, &diagonalMatrix);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create deqOffset aclTensor.
ret = CreateAclTensor(deqOffsetHostData, deqOffsetShape, &deqOffsetDeviceAddr, aclDataType::ACL_INT32, &deqOffset);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create deqScale aclTensor.
ret = CreateAclTensor(deqScaleHostData, deqScaleShape, &deqScaleDeviceAddr, aclDataType::ACL_UINT64, &deqScale);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the out aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT, &out);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the outFp16 aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outFp16DeviceAddr, aclDataType::ACL_FLOAT16, &outFp16);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// 3. Call the CANN operator library API. Replace with the specific API name.
// Example of calling the aclnnWeightQuantBatchMatmul API.
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// aclnn cast fp16
//x1
ret = aclnnCastGetWorkspaceSize(x1, aclDataType::ACL_FLOAT16, x1Fp16, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
ret = aclnnCast(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast failed. ERROR: %d\n", ret); return ret);
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// addOffset
ret = aclnnCastGetWorkspaceSize(addOffset, aclDataType::ACL_FLOAT16, addOffsetFp16, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
ret = aclnnCast(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast failed. ERROR: %d\n", ret); return ret);
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// mulScale
ret = aclnnCastGetWorkspaceSize(mulScale, aclDataType::ACL_FLOAT16, mulScaleFp16, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
ret = aclnnCast(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast failed. ERROR: %d\n", ret); return ret);
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// Call the first-phase API of aclnnWeightQuantBatchMatmul.
ret = aclnnWeightQuantBatchMatmulGetWorkspaceSize(x1Fp16, x2, diagonalMatrix, deqOffset, deqScale, addOffsetFp16, mulScaleFp16, nullptr, transposeX1, transposeX2, antiquantScale, antiquantOffset, outFp16, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnWeightQuantBatchMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnWeightQuantBatchMatmul.
ret = aclnnWeightQuantBatchMatmul(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnWeightQuantBatchMatmul failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronize and wait for task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// fp16 to fp 32
ret = aclnnCastGetWorkspaceSize(outFp16, aclDataType::ACL_FLOAT, out, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCastGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
ret = aclnnCast(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCast failed. ERROR: %d\n", ret); return ret);
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device memory to the host side. Modify this based on the specific API definition.
auto size = GetShapeSize(outShape);
std::vector<float> resultData(size, 0);
ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
}
// 6. Release aclTensor and aclScalar. Modify this based on the specific API definition.
aclDestroyTensor(x1);
aclDestroyTensor(x1Fp16);
aclDestroyTensor(x2);
aclDestroyTensor(addOffset);
aclDestroyTensor(addOffsetFp16);
aclDestroyTensor(mulScaleFp16);
aclDestroyTensor(mulScale);
aclDestroyTensor(out);
aclDestroyTensor(outFp16);
// 7. Release device resources. Modify based on the specific API definition.
aclrtFree(x1DeviceAddr);
aclrtFree(x2DeviceAddr);
aclrtFree(addOffsetDeviceAddr);
aclrtFree(deqScaleDeviceAddr);
aclrtFree(mulScaleDeviceAddr);
aclrtFree(diagonalMatrixDeviceAddr);
aclrtFree(deqOffsetDeviceAddr);
aclrtFree(outDeviceAddr);
aclrtFree(x1Fp16DeviceAddr);
aclrtFree(addOffsetFp16DeviceAddr);
aclrtFree(mulScaleFp16DeviceAddr);
aclrtFree(outFp16DeviceAddr);
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
free(diagonalMatrixData);
free(deqOffsetData);
free(deqScaleData);
return 0;
}