aclnnQuantMatmul
Note: This API will be deprecated in later versions. Use the latest aclnnQuantMatmulV5 API instead.
Applicable Products
| Product | Supported |
|---|---|
| × | |
| √ | |
| √ | |
| × | |
| × | |
| × |
Function
Description: Performs quantized matrix multiplication. The minimum supported dimension is 2, and the maximum supported input dimension is 3. Similar APIs include aclnnMm (matrix multiplication that supports only 2D tensors as input) and aclnnBatchMatMul (3D matrix multiplication where the first dimension is the batch dimension).
Formula:
Function Prototype
Each operator is divided into a two-phase API. The aclnnQuantMatmulGetWorkspaceSize API must be called first to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow. Then, the aclnnQuantMatmul API is called to perform the computation.
aclnnStatus aclnnQuantMatmulGetWorkspaceSize(
const aclTensor *x1,
const aclTensor *x2,
const aclTensor *bias,
float deqScale,
aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnQuantMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
const aclrtStream stream)aclnnQuantMatmulGetWorkspaceSize
Parameters
Return value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 The input x1, x2, or out is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type, data format, or dimension of x1, x2, bias, or out is not within the supported range. The data types of x1 and x2 cannot be used for data type deduction. The input shapes of x1 and x2 do not satisfy the matrix multiplication relationship. The shapes of x2 and bias are inconsistent. The bias exists and is an empty tensor where m and n are not 0 but k is 0.
aclnnQuantMatmul
Parameters:
Parameter Input/Output Description workspace Input Specifies the workspace memory address allocated on the device side. workspaceSize Input Specifies the workspace size allocated on the device side, obtained from the first-phase API aclnnQuantMatmulGetWorkspaceSize. executor Input Operator executor, which contains the operator computation flow. stream Input Specifies the stream for task execution. Return value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic description:
Atlas A3 training products/Atlas A3 inference products ,Atlas A2 training products/Atlas A2 inference products : aclnnQuantMatmul defaults to a deterministic implementation.
To migrate this API to the aclnnQuantMatmulV4 API:
The inputs x1, x2, and bias can be directly mapped to x1, x2, and bias in the aclnnQuantMatmulV4 API.
The input deqScale is of FLOAT type. Construct this FLOAT value into a FLOAT-type aclTensor with a shape of (1,) (see Example for CreateAclTensor), then use aclnnTransQuantParamV2 to convert it into a uint64_t aclTensor with a shape of (1,) (see aclnnQuantMatmulV4 Example), denoted as scale, which corresponds to the scale parameter in the aclnnQuantMatmulV4 API.
In the aclnnQuantMatmulV4 API, the optional inputs offset and pertokenScaleOptional are set to nullptr, and both transposeX1 and transposeX2 are set to false.
The API parameters are set to
aclnnQuantMatmulV4GetWorkspaceSize(x1, x2, scale, nullptr, nullptr, bias, false, false, out, workspaceSize, executor).
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include <memory>
#include "acl/acl.h"
#include "aclnnop/aclnn_quant_matmul.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define CHECK_FREE_RET(cond, return_expr) \
do { \
if (!(cond)) { \
Finalize(deviceId, stream);\
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate: resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate memory on the device side.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from the host side to the device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Compute the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
void Finalize(int32_t deviceId, aclrtStream stream) {
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
}
int aclnnQuantMatmulTest(int32_t deviceId, aclrtStream &stream) {
auto ret = Init(deviceId, &stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. Custom construction is required based on the API.
std::vector<int64_t> x1Shape = {2, 2};
std::vector<int64_t> x2Shape = {2, 2};
std::vector<int64_t> biasShape = {2};
std::vector<int64_t> outShape = {2, 2};
void* x1DeviceAddr = nullptr;
void* x2DeviceAddr = nullptr;
void* biasDeviceAddr = nullptr;
void* outDeviceAddr = nullptr;
aclTensor* x1 = nullptr;
aclTensor* x2 = nullptr;
aclTensor* bias = nullptr;
aclTensor* out = nullptr;
std::vector<int8_t> x1HostData{1, 1, 1, 1};
std::vector<int8_t> x2HostData{1, 1, 1, 1};
std::vector<int32_t> biasHostData{1, 1};
std::vector<uint16_t> outHostData{1, 1, 1, 1}; // Actually in float16 half-precision format.
// Create the x1 aclTensor.
ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
CHECK_FREE_RET(ret == ACL_SUCCESS, return ret);
// Create the other aclTensor.
ret = CreateAclTensor(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2TensorPtr(x2, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x2DeviceAddrPtr(x2DeviceAddr, aclrtFree);
CHECK_FREE_RET(ret == ACL_SUCCESS, return ret);
ret = CreateAclTensor(biasHostData, biasShape, &biasDeviceAddr, aclDataType::ACL_INT32, &bias);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> biasTensorPtr(bias, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> biasDeviceAddrPtr(biasDeviceAddr, aclrtFree);
CHECK_FREE_RET(ret == ACL_SUCCESS, return ret);
// Create the out aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
CHECK_FREE_RET(ret == ACL_SUCCESS, return ret);
float deqScale = 1.0f;
// 3. Call the CANN operator library API. Replace with the specific API name.
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// Call the first-phase API of aclnnQuantMatmul.
ret = aclnnQuantMatmulGetWorkspaceSize(x1, x2, bias, deqScale, out, &workspaceSize, &executor);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
void* workspaceAddr = nullptr;
std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
workspaceAddrPtr.reset(workspaceAddr);
}
// Call the second-phase API of aclnnQuantMatmul.
ret = aclnnQuantMatmul(workspaceAddr, workspaceSize, executor, stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmul failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Wait for the task execution to complete synchronously.
ret = aclrtSynchronizeStream(stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
auto size = GetShapeSize(outShape);
std::vector<float> resultData(size, 0);
ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
}
return ACL_SUCCESS;
}
int main() {
// 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
// Set deviceId based on your actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = aclnnQuantMatmulTest(deviceId, stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulTest failed. ERROR: %d\n", ret); return ret);
Finalize(deviceId, stream);
return 0;
}