aclnnQuantMatmulReduceSumWeightNz
Applicable Products
| Product | Supported |
|---|---|
| × | |
| √ | |
| √ | |
| × | |
| × | |
| × |
Function
Description: Performs quantized grouped matrix computation, and then sums the matrix computation results of all groups and outputs the result.
Formula:
Function Prototype
Each operator is divided into a two-phase API. You must first call the "aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize" API to obtain the workspace size required for computation and an executor that includes the operator's computation process, and then call the "aclnnQuantMatmulReduceSumWeightNz" API to perform the computation.
aclnnStatus aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize(
const aclTensor *x1,
const aclTensor *x2,
const aclTensor *x1Scale,
const aclTensor *x2Scale,
const aclTensor *yScale,
const aclTensor *x1Offset,
const aclTensor *x2Offset,
const aclTensor *yOffset,
const aclTensor *bias,
bool transposeX1,
bool transposeX2,
int64_t groupSize,
const aclIntArray *dims,
bool keepDims,
aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnQuantMatmulReduceSumWeightNz(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize
Parameters
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor x1 (aclTensor*) Input x1 in the formula. Null not supported for the tensor. INT8 ND (batch, m, k) - x2 (aclTensor*) Input In the formula, x2. - Null tensor is not supported.
- Each dimension indicates: (batch, n1, k1, k0, n0), where k0 = 16, n0 = 32. k in the x1 shape and k1 in the x2 shape must satisfy: ceil(k / 16) = k1. n1 in the x2 shape and n in out must satisfy: ceil(n / n0) = n1.
- The aclnnCalculateMatmulWeightSizeV2 API and aclnnTransMatmulWeight API can be used to convert the input format from ND to an AI processor-friendly data layout. The original ND format shape is (batch, k, n).
INT8 NZ 5D - x1Scale (aclTensor*) Input x1Scale in the formula. - Null not supported.
- During actual computation, x1Scale is broadcast to (batch, m, n).
FLOAT32 ND (batch, m) √ x2Scale (aclTensor*) Input x2Scale in the formula. - Null tensor not supported.
- During actual computation, x2Scale is broadcast to (batch, m, n).
BFLOAT16 ND (n,) √ yScale (aclTensor*) Input Reserved parameter. Not supported in the current version. Nullptr must be passed. - - - - x1Offset (aclTensor*) Input Reserved parameter. Not supported in the current version. Nullptr must be passed. - - - - x2Offset（aclTensor*） Input. Reserved Parameter. Not supported in the current version. Must be passed as nullptr. - - - - yOffset (aclTensor*) Input Reserved parameter. Not supported in the current version. Must be passed as nullptr. - - - - bias (aclTensor*) Input Reserved parameter. Not supported in the current version. Must be passed as nullptr. - - - - transposeX1 (bool) Input Indicates whether the input shape of x1 includes transpose. The current version only supports false, which indicates that the input shape of x1 remains unchanged. bool - - - transposeX2 (bool) Input Indicates whether the input shape of x2 includes transpose. The current version only supports false, indicating that the input shape meaning of x2 remains unchanged. bool - - - groupSize（int64_t） Input Reserved Parameter. Not supported in the current version. 0 must be passed. - - - - dims (aclIntArray *) Input Specifies the reduce dimension. In the current version, only [0] is supported, indicating ReduceSum is performed on dimension 0 (batch dimension). INT64 - - - keepDims (bool) Input Indicates whether to retain the dimensions of the input tensor in the output tensor. Only false is supported in the current version. - - - - out (aclTensor*) Output Out in the formula. - BFLOAT16 ND (m, n) √ workspaceSize (uint64_t*) Output Returns the workspace size to be applied by the device side. - - - - - executor (aclOpExecutor**) Output Returns the operator executor, which includes the operator calculation process. - - - - - Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Return Code Description ACLNN_ERR_PARAM_NULLPTR 161001 x1, x2, x1Scale, x2Scale, or out is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type or data format of x1, x2, x1Scale, x2Scale, or out is not within the supported range. The shape of x1, x2, x1Scale, x2Scale, or out does not meet the validation conditions. x1, x2, x1Scale, x2Scale, or out is an empty tensor.
aclnnQuantMatmulReduceSumWeightNz
Parameters
Parameter Input/Output Description workspace Input Workspace memory address applied by the Device side. workspaceSize Input Size of the workspace applied by the device side, obtained through the first-phase API aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize. executor Input Operator executor that includes the calculation process. stream Input Specifies the stream for task execution. Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic description:
- aclnnQuantMatmulReduceSumWeightNz defaults to a non-deterministic implementation. Deterministic computation can be enabled through aclrtCtxSetSysParamOpt.
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <memory>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_permute.h"
#include "aclnnop/aclnn_quant_matmul_weight_nz.h"
#include "aclnnop/aclnn_trans_matmul_weight.h"
#include "aclnnop/aclnn_trans_quant_param_v2.h"
#include "aclnnop/aclnn_quant_matmul_reduce_sum.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define CHECK_FREE_RET(cond, return_expr) \
do { \
if (!(cond)) { \
Finalize(deviceId, stream); \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t> &shape)
{
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream *stream)
{
// Boilerplate: resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
aclDataType dataType, aclTensor **tensor)
{
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to apply for device-side memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy host-side data to device-side memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call aclCreateTensor to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
void Finalize(int32_t deviceId, aclrtStream stream)
{
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
}
template <typename T>
int CreateAclTensorX2(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
aclDataType dataType, aclTensor **tensor)
{
auto size = static_cast<uint64_t>(GetShapeSize(shape));
const aclIntArray *mat2Size = aclCreateIntArray(shape.data(), shape.size());
auto ret = aclnnCalculateMatmulWeightSizeV2(mat2Size, dataType, &size);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSizeV2 failed. ERROR: %d\n", ret);
return ret);
size *= sizeof(T);
// Call aclrtMalloc to apply for device-side memory.
ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Copy host-side data to device-side memory using aclrtMemcpy.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
std::vector<int64_t> storageShape;
storageShape.push_back(GetShapeSize(shape));
// Call aclCreateTensor to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_FRACTAL_NZ,
storageShape.data(), storageShape.size(), *deviceAddr);
return 0;
}
int aclnnQuantMatmulWeightNzTest(int32_t deviceId, aclrtStream &stream)
{
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct inputs and outputs. Custom construction is required based on the API.
int64_t b = 8;
int64_t m = 2048;
int64_t k = 1024;
int64_t n = 7168;
// Create x1 aclTensor.
std::vector<int64_t> x1Shape = {b, m, k};
void *x1DeviceAddr = nullptr;
aclTensor *x1 = nullptr;
std::vector<int8_t> x1HostData(b * m * k, 1);
ret = CreateAclTensor(x1HostData, x1Shape, &x1DeviceAddr, aclDataType::ACL_INT8, &x1);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1TensorPtr(x1, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x1DeviceAddrPtr(x1DeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x2 aclTensor in AI processor affinity data layout format.
std::vector<int64_t> x2Shape = {b, k, n};
void *x2DeviceAddr = nullptr;
aclTensor *x2 = nullptr;
std::vector<int8_t> x2HostData(b * k * n, 1);
ret = CreateAclTensorX2(x2HostData, x2Shape, &x2DeviceAddr, aclDataType::ACL_INT8, &x2);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2HPTensorPtr(x2, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x2HPDeviceAddrPtr(x2DeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x1Scale aclTensor.
std::vector<int64_t> x1ScaleShape = {b, m};
void *x1ScaleDeviceAddr = nullptr;
std::vector<float> x1ScaleHostData(b * m, 1);
aclTensor *x1Scale = nullptr;
ret = CreateAclTensor(x1ScaleHostData, x1ScaleShape, &x1ScaleDeviceAddr, aclDataType::ACL_FLOAT, &x1Scale);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x1ScaleTensorPtr(x1Scale, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x1ScaleDeviceAddrPtr(x1ScaleDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x2Scale aclTensor.
std::vector<int64_t> x2ScaleShape = {n};
void *x2ScaleDeviceAddr = nullptr;
aclTensor *x2Scale = nullptr;
std::vector<uint16_t> x2ScaleHostData(n, 1); // Actually in bfloat16 half-precision format.
ret = CreateAclTensor(x2ScaleHostData, x2ScaleShape, &x2ScaleDeviceAddr, aclDataType::ACL_BF16, &x2Scale);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> x2ScaleTensorPtr(x2Scale, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> x2ScaleDeviceAddrPtr(x2ScaleDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the out aclTensor.
std::vector<int64_t> outShape = {m, n};
void *outDeviceAddr = nullptr;
aclTensor *out = nullptr;
std::vector<uint16_t> outHostData(m * n, 1); // Actually in bfloat16 half-precision format.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_BF16, &out);
std::unique_ptr<aclTensor, aclnnStatus (*)(const aclTensor *)> outTensorPtr(out, aclDestroyTensor);
std::unique_ptr<void, aclError (*)(void *)> outDeviceAddrPtr(outDeviceAddr, aclrtFree);
CHECK_RET(ret == ACL_SUCCESS, return ret);
bool transposeX1 = false;
bool transposeX2 = false;
// Create the dims aclIntArray.
std::vector<int64_t> dimsData = {0};
aclIntArray *dims = nullptr;
dims = aclCreateIntArray(dimsData.data(), dimsData.size());
CHECK_RET(dims != nullptr, return ret);
// 3. Call the CANN operator library API. Replace with the specific API name.
uint64_t workspaceSize = 0;
aclOpExecutor *executor = nullptr;
// Call the first-phase API of aclnnQuantMatmulReduceSumWeightNz.
ret = aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize(
x1, x2, x1Scale, x2Scale, nullptr, nullptr, nullptr, nullptr, nullptr, transposeX1, transposeX2, 0,
dims, false, out, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulReduceSumWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret);
return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
void *workspaceAddr = nullptr;
std::unique_ptr<void, aclError (*)(void *)> workspaceAddrPtr(nullptr, aclrtFree);
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
workspaceAddrPtr.reset(workspaceAddr);
}
// Call the second-phase API of aclnnQuantMatmulReduceSumWeightNz.
ret = aclnnQuantMatmulReduceSumWeightNz(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulReduceSumWeightNz failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronously wait for task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device memory to the host memory. Modify based on the specific API definition.
auto size = GetShapeSize(outShape);
std::vector<uint16_t> resultData(
size, 0); // In C, bfloat16 data cannot be printed directly. Read it as uint16 and manually convert it to fp16 via binary.
ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret);
return ret);
for (int64_t i = 0; i < 5; i++) {
LOG_PRINT("result[%ld] is: %u\n", i, resultData[i]);
}
return ACL_SUCCESS;
}
int main()
{
// 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
// Fill in the deviceId based on your actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = aclnnQuantMatmulWeightNzTest(deviceId, stream);
CHECK_FREE_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantMatmulWeightNzTest failed. ERROR: %d\n", ret);
return ret);
Finalize(deviceId, stream);
return 0;
}