aclnnGroupedMatmul
Notice: This API will be deprecated in later versions. Use the latest aclnnGroupedMatmulV5 API instead.
Applicable Products
| Product | Supported |
|---|---|
| √ | |
| √ | |
| √ |
Function
Description: Implements grouped matrix multiplication, where the dimension sizes of each group can differ. The basic function is matrix multiplication, as in , where g is the number of groups and are the corresponding shapes. The following four scenarios are supported based on the number of tensors for x, weight, and y:
x, weight, and y are all multi-tensor, meaning the tensor corresponding to each group's data is independent.
x is a single tensor, while weight and y are multiple tensors. In this case, use the optional parameter group_list to specify how x is grouped along the row dimension. For example, group_list[0]=10 indicates that the first 10 rows of x participate in the first group of matrix multiplication.
x and weight are multiple tensors, while y is a single tensor. In this case, the results of each group of matrix multiplication are stored contiguously in the same tensor.
x and y are single tensors, while weight is multiple tensors. This is a combination of the previous two cases.
NOTE A single tensor means that all grouped tensors in a tensor list are merged into one along the M axis; otherwise, they are multiple tensors.
Formula:
Non-quantized Scenario:
Quantized Scenario:
Dequantized Scenario:
Pseudo-quantized Scenario:
Function Prototype
Each operator is divided into a two-phase API. You must first call the "aclnnGroupedMatmulGetWorkspaceSize" API to obtain the input parameters and calculate the required workspace size based on the process, and then call the "aclnnGroupedMatmul" API to perform the computation.
aclnnStatus aclnnGroupedMatmulGetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *biasOptional,
const aclTensorList *scaleOptional,
const aclTensorList *offsetOptional,
const aclTensorList *antiquantScaleOptional,
const aclTensorList *antiquantOffsetOptional,
const aclIntArray *groupListOptional,
int64_t splitItem,
const aclTensorList *y,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnGroupedMatmul(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnGroupedMatmulGetWorkspaceSize
Parameters:
Atlas A2 training products/Atlas A2 inference products :x and weight support FLOAT16, BFLOAT16, and INT8.
y supports FLOAT16, BFLOAT16, INT8, and FLOAT32.
Ascend 950PR/Ascend 950DT :x supports FLOAT16, BFLOAT16, and FLOAT32.
weight supports FLOAT16, BFLOAT16, FLOAT32, and INT8.
y supports FLOAT16, BFLOAT16, and FLOAT32.
Does not support scaleOptional or offsetOptional.
Return value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 Indicates that a required input, output, or required attribute is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type and data format of x, weight, biasOptional, scaleOptional, offsetOptional, antiQuantScaleOptional, antiQuantOffsetOptional, groupListOptional, splitItem, and y are not within the supported range. The length of weight is not within the supported range. If bias is not empty, the length of bias is not equal to the length of weight. In scenarios where splitItem is 0 or 1, the length of y is not equal to the length of weight, and the length of groupListOptional is not equal to the length of weight. Scenarios where splitItem is 2 or 3, and the length of y is not equal to 1.
aclnnGroupedMatmul
Parameters:
Parameter Input/Output Description workspace Input Specifies the workspace memory address applied on the device side. workspaceSize Input Specifies the workspace size applied on the device side, which is obtained through the first-phase API aclnnGroupedMatmulGetWorkspaceSize. executor Input Operator executor that contains the operator computation flow. stream Input Specifies the stream for task execution. Return value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic computation:
aclnnGroupedMatmul defaults to a deterministic implementation.
Atlas A2 training products/Atlas A2 inference products :The supported input types in the non-quantized scenario are:
x is FLOAT16, weight is FLOAT16, biasOptional is FLOAT16, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, and y is FLOAT16.
x is BFLOAT16, weight is BFLOAT16, biasOptional is FLOAT32, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, and y is BFLOAT16.
Supported input types in the Quantized Scenario:
- x is INT8, weight is INT8, biasOptional is INT32, scaleOptional is UINT64, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, and y is INT8.
Supported input types in the Pseudo-quantized Scenario:
x is FLOAT16, weight is INT8, biasOptional is FLOAT16, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is FLOAT16, antiquantOffsetOptional is FLOAT16, and y is FLOAT16.
x is BFLOAT16, weight is INT8, biasOptional is FLOAT32, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is BFLOAT16, antiquantOffsetOptional is BFLOAT16, and y is BFLOAT16.
If groupListOptional is passed, groupListOptional must be a non-negative increasing sequence, and the length of groupListOptional cannot be 1.
Currently supported scenarios:
In the supported scenarios, "single" indicates a single tensor and "multiple" indicates multiple tensors. The order is x, weight, y. For example, "single-multiple-single" indicates that x is a single tensor, weight consists of multiple tensors, and y is a single tensor.
Supported Scenarios Scenario Constraints Multi-Multi-Multi 1) Only supports splitItem being 0/1.
2) Tensors in x must have the same number of dimensions, supporting 2 to 6 dimensions. Tensors in weight must be 2-dimensional. Tensors in y must have the same number of dimensions as those in x.
3) If any tensor in x has more than 2 dimensions, groupListOptional must be empty.
4) If tensors in x are 2-dimensional and groupListOptional is passed, the differences of groupListOptional must have a one-to-one correspondence with the first dimension of the tensors in x.Single-Multi-Single 1) Only supports splitItem being 2/3.
2) groupListOptional must be passed, and its last value must equal the first dimension of the tensor in x.
3) Tensors in x, weight, and y must be 2-dimensional.
4) The N axis of each tensor in weight must be the same.Single-Multi-Multi 1) Only supports splitItem being 0/1.
2) groupListOptional must be passed, and the differences of groupListOptional must have a one-to-one correspondence with the first dimension of the tensors in y.
3) Tensors in x, weight, and y must be 2-dimensional.Multi-Multi-Single 1) Only supports splitItem being 2/3.
2) Tensors in x, weight, and y must be 2-dimensional.
3) The N axis of each tensor in weight must be the same.
4) If groupListOptional is passed, the differences of groupListOptional must have a one-to-one correspondence with the first dimension of the tensors in x.The size of the last dimension of each group of tensors in x and weight must be less than 65536. The last dimension of refers to the K axis of when the attribute transpose_x is false, or the M axis of when transpose_x is true. The last dimension of refers to the N axis of when the attribute transpose_weight is false, or the K axis of when transpose_weight is true.
The size of each dimension of each group of tensors in x and weight, after 32-byte alignment, must be less than the maximum int32 value 2147483647.
Ascend 950PR/Ascend 950DT :Non-quantized Scenario Constraints
The data types supported in the non-quantized scenario are:
If groupListOptional is passed, it must be a non-negative increasing sequence, and its length cannot be 1.
The following input parameters are empty: scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional.
The supported data type combinations for non-empty parameters must satisfy the following table:
x weight biasOptional y BFLOAT16 BFLOAT16 BFLOAT16/FLOAT32/null BFLOAT16 FLOAT16 FLOAT16 FLOAT16/FLOAT32/null FLOAT16
Pseudo-quantized Scenario Constraints
The data types supported in the pseudo-quantized scenario are as follows:
The following input parameters are empty: scaleOptional, offsetOptional.
The supported data type combinations for non-empty parameters must satisfy the following table:
x weight biasOptional antiquantScaleOptional antiquantOffsetOptional y BFLOAT16 INT8 BFLOAT16/FLOAT32/null BFLOAT16 BFLOAT16 BFLOAT16 FLOAT16 INT8 FLOAT16/null FLOAT16 FLOAT16 FLOAT16 antiquantScaleOptional and non-empty biasOptional and antiquantOffsetOptional must satisfy the following table:
Usage Scenarios Shape Constraint weight multi-tensor Each tensor is 1D with shape (). It is not allowed that some tensors in a tensorList have shape () while others are empty.
Only supports the multi-multi-multi scenario.
Supported Scenario Constraints
- In supported scenarios, "single" indicates a single tensor, "multiple" indicates multiple tensors, and the order is x, weight, y. For example, "single-multiple-single" indicates a scenario where x is a single tensor, weight consists of multiple tensors, and y is a single tensor. | Supported Scenario | Scenario Restriction | |:-------:| :-------| | multiple-multiple-multiple |1) Only supports splitItem 0/1.<br>2) In the pseudo-quantized scenario, tensors in x must have the same dimensions, supporting 2 to 6 dimensions, and tensors in y must have the same dimensions as x. In the non-quantized scenario, tensors in x and y must be 2-dimensional with shapes ($m_i$, $k_i$) and ($m_i$, $n_i$) respectively. Tensors in weight must be 2-dimensional with shape ($n_i$, $k_i$) or ($k_i$, $n_i$). Tensors in bias must be 1-dimensional with shape ($n_i$).<br>3) If any tensor in x has more than 2 dimensions, groupListOptional must be empty.<br>4) If tensors in x are 2-dimensional and groupListOptional is passed, the differences of groupListOptional must have a one-to-one correspondence with the first dimension of tensors in x, and the maximum length is 128.<br>5) Only supports ND in, ND out.<br>6) Does not support x transposition and does not support weight transposition. | | single-multiple-single |1) Only supports splitItem 2/3.<br>2) groupListOptional must be passed, the last value must equal the first dimension of the tensor in x, and the maximum length is 128.<br>3) Tensors in x and y must be 2-dimensional with shapes (M, K) and (M, N) respectively. Tensors in weight must be 2-dimensional with shape (N, K) or (K, N). Tensors in bias must be 1-dimensional with shape (N).<br>4) The N axis of each tensor in weight must be the same.<br>5) Only supports ND in, ND out.<br>6) Only supports non-quantized scenarios.<br>7) Does not support x transposition. Supports weight transposition (if weight consists of multiple tensors, whether each tensor is transposed must be consistent).| | single-multiple-multiple |1) Only supports splitItem 0/1.<br>2) groupListOptional must be passed, the last value must equal the first dimension of the tensor in x, and the maximum length is 128.<br>3) Tensors in x and y must be 2-dimensional with shapes (M, K) and (M, N) respectively. Tensors in weight must be 2-dimensional with shape (N, K) or (K, N). Tensors in bias must be 1-dimensional with shape (N).<br>4) Only supports ND in, ND out.<br>5) Only supports non-quantized scenarios.<br>6) Does not support x transposition. Supports weight transposition (if weight consists of multiple tensors, whether each tensor is transposed must be consistent).| | multiple-multiple-single |1) Only supports splitItem 2/3.<br>2) Tensors in x and y must be 2-dimensional with shapes (M, K) and (M, N) respectively. Tensors in weight must be 2-dimensional with shape (N, K) or (K, N). Tensors in bias must be 1-dimensional with shape (N).<br>3) The N axis of each tensor in weight must be the same.<br>4) If groupListOptional is passed, the differences of groupListOptional must have a one-to-one correspondence with the first dimension of tensors in x, and the maximum length is 128.<br>5) Only supports ND in, ND out.<br>6) Only supports non-quantized scenarios.<br>7) Does not support x transposition. Supports weight transposition (if weight consists of multiple tensors, whether each tensor is transposed must be consistent).|
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate. Initialize resources.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate memory on the device side.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from the host side to the device memory.
std::vector<T> hostData(GetShapeSize(shape), 0);
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
int CreateAclTensorList(const std::vector<std::vector<int64_t>>& shapes, void** deviceAddr,
aclDataType dataType, aclTensorList** tensor) {
int size = shapes.size();
std::vector<aclTensor*> tensors(size);
for (int i = 0; i < size; i++) {
int ret = CreateAclTensor<uint16_t>(shapes[i], deviceAddr + i, dataType, tensors.data() + i);
CHECK_RET(ret == ACL_SUCCESS, return ret);
}
*tensor = aclCreateTensorList(tensors.data(), size);
return ACL_SUCCESS;
}
int main() {
// 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
// Enter the deviceId based on the actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
// Handle the check result as needed.
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. Customize the construction based on the API interface.
std::vector<std::vector<int64_t>> xShape = {{1, 16}, {4, 32}};
std::vector<std::vector<int64_t>> weightShape= {{16, 24}, {32, 16}};
std::vector<std::vector<int64_t>> biasShape = {{24}, {16}};
std::vector<std::vector<int64_t>> yShape = {{1, 24}, {4, 16}};
void* xDeviceAddr[2];
void* weightDeviceAddr[2];
void* biasDeviceAddr[2];
void* yDeviceAddr[2];
aclTensorList* x = nullptr;
aclTensorList* weight = nullptr;
aclTensorList* bias = nullptr;
aclIntArray* groupedList = nullptr;
aclTensorList* scale = nullptr;
aclTensorList* offset = nullptr;
aclTensorList* antiquantScale = nullptr;
aclTensorList* antiquantOffset = nullptr;
aclTensorList* y = nullptr;
int64_t splitItem = 0;
// Create the x aclTensorList.
ret = CreateAclTensorList(xShape, xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the weight aclTensorList.
ret = CreateAclTensorList(weightShape, weightDeviceAddr, aclDataType::ACL_FLOAT16, &weight);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the bias aclTensorList.
ret = CreateAclTensorList(biasShape, biasDeviceAddr, aclDataType::ACL_FLOAT16, &bias);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the y aclTensorList.
ret = CreateAclTensorList(yShape, yDeviceAddr, aclDataType::ACL_FLOAT16, &y);
CHECK_RET(ret == ACL_SUCCESS, return ret);
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// 3. Call the CANN operator library API.
// Call the first-phase API of aclnnGroupedMatmul.
ret = aclnnGroupedMatmulGetWorkspaceSize(x, weight, bias, scale, offset, antiquantScale, antiquantOffset, groupedList, splitItem, y, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnGroupedMatmul.
ret = aclnnGroupedMatmul(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmul failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronize and wait for task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output values and copy the results from the Device memory to the Host memory. Modify based on the specific API definition.
for (int i = 0; i < 2; i++) {
auto size = GetShapeSize(yShape[i]);
std::vector<uint16_t> resultData(size, 0);
ret = aclrtMemcpy(resultData.data(), size * sizeof(resultData[0]), yDeviceAddr[i],
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t j = 0; j < size; j++) {
LOG_PRINT("result[%ld] is: %hu\n", j, resultData[j]);
}
}
// 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
aclDestroyTensorList(x);
aclDestroyTensorList(weight);
aclDestroyTensorList(bias);
aclDestroyTensorList(y);
// 7. Release device resources. Modify based on the specific API definition.
for (int i = 0; i < 2; i++) {
aclrtFree(xDeviceAddr[i]);
aclrtFree(weightDeviceAddr[i]);
aclrtFree(biasDeviceAddr[i]);
aclrtFree(yDeviceAddr[i]);
}
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}