aclnnGroupedMatmulV3
Notice: This API will be deprecated in later versions. Use the latest aclnnGroupedMatmulV5 API instead.
Applicable Products
| Product | Supported |
|---|---|
| √ | |
| √ | |
| √ | |
| × | |
| √ | |
| × |
Function
Description: Performs grouped matrix multiplication, where the dimension sizes of each group can differ. The basic function is matrix multiplication, as in , where g is the number of groups and are the corresponding shapes. Both input and output data types are aclTensorList, and the corresponding functions are:
k-axis grouping: values differ across groups, but are the same for each group. In this case, can be concatenated along .
m-axis grouping: is the same across groups, and can be concatenated along .
Compared with the GroupedMatmul API, this API adds:
Atlas A2 training products/Atlas A2 inference products :In non-quantization scenarios, weight transpose is supported (transpose refers to the scenario where, if the shape is [M,K], the stride is [1,M] and the data layout is [K,M]).
Supports m-axis and k-axis grouping, indicated by groupType.
When x, weight, and y are all single tensors, in non-quantization scenarios, both x and weight inputs of float32 type are supported.
In quantization and pseudo-quantization scenarios, weight transposition is supported, and weight as a single tensor is supported.
For features supported by the aclnnGroupedMatmulGetWorkspaceSize API, this API does not support the scenario where x is a single tensor and weight/y are multiple tensors.
Ascend 950PR/Ascend 950DT :In pseudo-quantization scenarios, weight transposition is supported, and x, weight, and y as single tensors are all supported.
For features supported by the aclnnGroupedMatmulGetWorkspaceSize API, this API does not support the scenario where x is a single tensor and weight/y are multiple tensors.
NOTE
A single tensor means that all grouped tensors in a tensor list are merged into one along the grouping axis specified by groupType; otherwise, they are multiple tensors.
Tensor transpose: refers to the scenario where the tensor shape is [M, K], the stride is [1, M], and the data layout is [K, M], i.e., a non-contiguous tensor.
Formula:
Non-quantization scenario:
Quantization scenario:
Dequantization scenario:
Pseudo-quantization scenario:
Function Prototype
Each operator is divided into a two-phase API. You must first call the "aclnnGroupedMatmulV3GetWorkspaceSize" API to obtain the input parameters and calculate the required workspace size based on the process, and then call the "aclnnGroupedMatmulV3" API to perform the computation.
aclnnStatus aclnnGroupedMatmulV3GetWorkspaceSize(
const aclTensorList *x,
const aclTensorList *weight,
const aclTensorList *biasOptional,
const aclTensorList *scaleOptional,
const aclTensorList *offsetOptional,
const aclTensorList *antiquantScaleOptional,
const aclTensorList *antiquantOffsetOptional,
const aclTensor *groupListOptional,
int64_t splitItem,
int64_t groupType,
const aclTensorList *y,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnGroupedMatmulV3(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnGroupedMatmulV3GetWorkspaceSize
Parameters:
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor x (aclTensorList) Input Input x in the formula. The tensorList length supports [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, FLOAT32, INT8 ND - - weight (aclTensorList) Input Weight in the formula. The tensorList length supports [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, FLOAT32, INT8 ND - - biasOptional (aclTensorList) Input Bias in the formula. Has the same length as weight. INT32, BFLOAT16, FLOAT16, FLOAT32 ND - - scaleOptional (aclTensorList) Input Represents the scaling factor in the quantization parameter. Generally, its length is the same as that of weight. UINT64, INT64 ND - - offsetOptional (aclTensorList) Input Represents the offset in the quantization parameters. Has the same length as weight. FLOAT32 ND - - antiquantScaleOptional (aclTensorList) Input Represents the scaling factor in the pseudo-quantization parameters. Has the same length as weight. FLOAT16, BFLOAT16 ND - - antiquantOffsetOptional (aclTensorList) Input Represents the offset in the pseudo-quantization parameters. Length same as weight. FLOAT16, BFLOAT16 ND - - groupListOptional (aclTensorList) Input Represents the matmul size distribution along the grouping axis for inputs and outputs. When the length of the output TensorList is 1, the last value in groupListOptional constrains the valid portion of the output data, and the unspecified portions in groupListOptional will not be updated. INT64 ND - - splitItem (int64_t) Input Integer parameter that specifies whether the output is split into multiple tensors. 0/1 indicates that the output consists of multiple tensors. 2/3 indicates that the output is a single tensor. - - - - groupType (int64_t) Input Integer parameter that specifies the axis to be grouped. If the matrix multiplication is C[m,n]=A[m,k]xB[k,n], groupType takes the following values: -1: no grouping, 0: group along the m axis, 2: group along the k axis. - - - - y (aclTensorList) Output Output y in the formula. The tensorList length supports [1, 128] or [1, 1024]. FLOAT16, BFLOAT16, INT8, FLOAT32, INT32 ND - - workspaceSize (uint64_t) Output Returns the workspace size to be allocated on the Device side. - - - - - executor Output Returns the op executor, which contains the operator computation flow. - - - - - Atlas A2 training products/Atlas A2 inference products :x supports FLOAT16, BFLOAT16, INT8, and FLOAT32.
weight supports FLOAT16, BFLOAT16, INT8, and FLOAT32.
biasOptional supports FLOAT16, FLOAT32, and INT32.
y supports FLOAT16, BFLOAT16, INT8, and FLOAT32.
The input parameters x and weight, and the output parameter y support up to 128 tensors.
Ascend 950PR/Ascend 950DT :x supports FLOAT16, BFLOAT16, FLOAT32, and INT8.
weight supports FLOAT16, BFLOAT16, FLOAT32, and INT8.
biasOptional supports FLOAT16, BFLOAT16, FLOAT32, and INT32.
y supports FLOAT16, BFLOAT16, FLOAT32, and INT8.
offsetOptional is not supported.
groupType supports grouping and non-grouping along the m-axis. Grouping along the k-axis is supported only in non-quantization scenarios.
The input parameters x and weight, and the output parameter y support a maximum of 1024 tensors in non-quantization scenarios, a maximum of 128 tensors in pseudo-quantization scenarios, and a maximum of 1 tensor in quantization scenarios.
Return value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 The input parameter is a required input, output, or required attribute and is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type or data format of x, weight, biasOptional, scaleOptional, offsetOptional, antiquantScaleOptional, antiquantOffsetOptional, groupListOptional, or out is not within the supported range. The length of weight is not within the supported range. If bias is not empty, the length of bias is not equal to the length of weight. The dimension of groupListOptional is 1. When splitItem is 2 or 3, the length of out is not equal to 1. When splitItem is 0 or 1, the length of out is not equal to the length of weight, or the length of groupListOptional is not equal to the length of weight.
aclnnGroupedMatmulV3
Parameters:
Parameters Input/Output Description workspace Input Workspace memory address applied for on the Device side. workspaceSize Input Workspace size applied for on the Device side, obtained from the first-phase API aclnnGroupedMatmulV3GetWorkspaceSize. executor Input Operator executor, which contains the operator computation flow. stream Input Specifies the stream that executes the task. Return value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic computation:
- aclnnGroupedMatmulV3 defaults to a deterministic implementation.
If groupListOptional is passed in, it must be a non-negative increasing sequence, and its length cannot be 1.
For each group of tensors in x and weight, the size of each dimension, after 32-byte alignment, must be less than the maximum int32 value 2147483647.
Atlas A2 training products/Atlas A2 inference products :Supported input types in non-quantization scenarios:
x is FLOAT16, weight is FLOAT16, biasOptional is FLOAT16, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, y is FLOAT16.
x is BFLOAT16, weight is BFLOAT16, biasOptional is FLOAT32, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, y is BFLOAT16.
x is FLOAT32, weight is FLOAT32, biasOptional is FLOAT32, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, y is FLOAT32 (supported only when x, weight, and y are all single-tensor scenarios).
Supported input types for the quantization scenario:
- x is INT8, weight is INT8, biasOptional is INT32, scaleOptional is UINT64, offsetOptional is empty, antiquantScaleOptional is empty, antiquantOffsetOptional is empty, and y is INT8.
Supported input types for the pseudo-quantization scenario:
x is FLOAT16, weight is INT8, biasOptional is FLOAT16, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is FLOAT16, antiquantOffsetOptional is FLOAT16, and y is FLOAT16.
x is BFLOAT16, weight is INT8, biasOptional is FLOAT32, scaleOptional is empty, offsetOptional is empty, antiquantScaleOptional is BFLOAT16, antiquantOffsetOptional is BFLOAT16, and y is BFLOAT16.
Supported scenarios for different groupType values:
The a16w8 and a16w4 scenarios support only groupType values of -1 and 0.
The A8W8, A8W4, and A4W4 scenarios support only the groupType 0 scenario where the number of x tensors is single.
The input types of x, weight, and y are aclTensorList, which represents an array object of the aclTensor type. In the following support scenario table, "single" indicates an aclTensorList consisting of one aclTensor, and "multiple" indicates an aclTensorList consisting of multiple aclTensors. For example, "single-multiple-single" means that x is a single tensor, weight is multiple tensors, and y is a single tensor.
groupType Number of x Tensors Number of weight Tensors Number of y Tensors splitItem groupListOptional Transpose Other Scenario Constraints -1 Multiple Multiple Multiple 0/1 groupListOptional must be null. 1) Transpose of x is not supported.
2) Transpose of weight is supported, but whether each tensor in the weight tensorList is transposed must be consistent.Tensors in x must have the same dimensions and support 2D. Tensors in weight must be 2D. Tensors in y must have the same dimensions as x. 0 Single Single Single 2/3 1) groupListOptional must be passed.
2) The first dimension of groupListOptional supports a maximum of 1024, that is, a maximum of 1024 groups.1) Transpose of x is not supported.
2) Transpose of weight is supported. Transpose of weight is not supported in A8W4 and A4W4 scenarios.Tensors in weight must be 3D. Tensors in x and y must be 2D. 0 Single Multiple Single 2/3 1) groupListOptional must be passed.
2) The first dimension of groupListOptional supports a maximum of 128, that is, a maximum of 128 groups.1) Transpose of x is not supported.
2) Transpose of weight is supported, but whether each tensor in the weight tensorList is transposed must be consistent.1) Tensors in x, weight, and y must be 2D.
2) The N axis of each tensor in weight must be equal.0 Multiple Multiple Single 2/3 1) groupListOptional is optional.
2) The first dimension of groupListOptional supports a maximum of 128, that is, a maximum of 128 groups.1) Transpose of x is not supported.
2) Transpose of weight is supported, but whether each tensor in the weight tensorList is transposed must be consistent.1) Tensors in x, weight, and y must be 2D.
2) The N axis of each tensor in weight must be equal.2 Single Single Single 2/3 1) groupListOptional must be passed.
2) The first dimension of groupListOptional supports a maximum of 1024, that is, a maximum of 1024 groups.1) x must be transposed.
2) weight cannot be transposed.1) Tensors in x and weight must be 2D, and tensors in y must be 3D.
2) bias must be null.2 Single Multiple Multiple 0/1 groupListOptional must be null. 1) x must be transposed.
2) weight cannot be transposed.1) Tensors in x, weight, and y must be 2D.
2) The length of weight supports a maximum of 128, that is, a maximum of 128 groups.
3) In the original shape, the sum of the first dimensions of all tensors in weight must not exceed the first dimension of x.
4) bias must be null.
The size of the last dimension of each group of tensors in x and weight should be less than 65536. The last dimension of refers to the K axis of when x is not transposed, or the M axis of when x is transposed. The last dimension of refers to the N axis of when weight is not transposed, or the K axis of when weight is transposed.
Ascend 950PR/Ascend 950DT :Non-Quantization Scenario Constraints
Data types supported in non-quantization scenarios:
The following input parameters are empty: scaleOptional, offsetOptional, antiQuantScaleOptional, antiQuantOffsetOptional
The supported data type combinations for non-empty parameters must satisfy the following table:
groupType x weight biasOptional y -1/0/2 BFLOAT16 BFLOAT16 BFLOAT16/FLOAT32/null BFLOAT16 -1/0/2 FLOAT16 FLOAT16 FLOAT16/FLOAT32/null FLOAT16 -1/0/2 FLOAT32 FLOAT32 FLOAT32/null FLOAT32
Pseudo-Quantization Scenario Constraints
The data types supported in the pseudo-quantization scenario are:
The following input parameters are empty: scaleOptional and offsetOptional.
The data type combinations supported by non-empty parameters must meet the following table:
groupType x weight antiquantScaleOptional antiquantOffsetOptional biasOptional y -1/0 BFLOAT16 INT8 BFLOAT16 BFLOAT16/null BFLOAT16/FLOAT32/null BFLOAT16 -1/0 FLOAT16 INT8 FLOAT16 FLOAT16/null FLOAT16/null FLOAT16 antiquantScaleOptional, non-empty biasOptional, and antiquantOffsetOptional must meet the following table (where g is the number of matmul groups, i.e., the number of groups):
groupType Use Scenario Shape Constraint -1 Weight multi-tensor Each tensor is 1D with shape (). It is not allowed that some tensors in a tensorList have shape () while others are empty. 0 Weight single-tensor Each tensor is 2D with shape (g, N). Only single-single-single and multi-multi-multi scenarios are supported.
Quantization Scenario Constraints
The following input parameters are empty: offsetOptional, antiquantScaleOptional, antiquantOffsetOptional.
Only the single-single-single scenario is supported.
The supported data type and dimension combinations for non-null parameters must satisfy the following table (where g is the number of matmul groups, i.e., the number of groups):
groupType x dtype x shape weight dtype weight shape biasOptional dtype biasOptional shape scaleOptional dtype scaleOptional shape out dtype out shape 0 INT8 (M,K) INT8 (g,K,N)/(g,N,K) INT32/null (g,N) UINT64/INT64 (g,N)/null INT8 (M,N)
groupType Scenario Constraints
Supported scenarios for different groupType values:
In supported scenarios, "single" indicates a single tensor, and "multiple" indicates multiple tensors. The order is x, weight, y. For example, "single-multiple-single" indicates a scenario where x is a single tensor, weight consists of multiple tensors, and y is a single tensor.
groupType Supported Scenario Scenario Restrictions -1 Multiple-multiple-multiple 1) Only splitItem 0/1 is supported.
2) For non-quantized x, tensors in out must be 2D with shapes (, ) and (, ) respectively. For pseudo-quantized scenarios, tensors in x must have the same number of dimensions, supporting 2–6 dimensions, and tensors in y must have the same number of dimensions as x. Tensors in weight must be 2D with shape (, ) or (, ). Tensors in bias must be 1D with shape ().
3) groupListOptional must be passed as null.
4) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must be consistent.
5) x does not support transposition.
6) Only ND in, ND out is supported.0 Single-single-single 1) Only splitItem 2/3 is supported.
2) Tensors in weight must be 3D with shape (g, N, K) or (g, K, N). Tensors in x and y must be 2D with shapes (M, K) and (M, N) respectively. Tensors in bias must be 2D with shape (g, N).
3) groupListOptional must be passed, and the last value must not exceed the first dimension of the tensor in x.
4) The first dimension of groupListOptional supports a maximum of 1024, that is, a maximum of 1024 groups.
5) Weight transposition is supported.
6) x does not support transposition.
7) Only ND in, ND out is supported.0 Single-multiple-single 1) Only splitItem 2/3 is supported.
2) groupListOptional must be passed, the last value must equal the first dimension of the tensor in x, and the length must not exceed 1024.
3) Tensors in x and y must be 2D with shapes (M, K) and (M, N) respectively. Tensors in weight must be 2D with shape (N, K) or (K, N). Tensors in bias must be 1D with shape (N).
4) The N axis of each tensor in weight must be equal.
5) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must be consistent.
6) x does not support transposition.
7) Only ND in, ND out is supported.
8) Only non-quantized scenarios are supported.0 Multiple-multiple-single 1) Only splitItem 2 is supported.
2) Tensors in x and y must be 2D with shapes (M, K) and (M, N) respectively. Tensors in weight must be 2D with shape (N, K) or (K, N). Tensors in bias must be 1D with shape (N).
3) The N axis of each tensor in weight must be equal.
4) If groupListOptional is passed, the differences of groupListOptional must correspond one-to-one with the first dimension of the tensor in x, and the length must not exceed 1024.
5) Weight transposition is supported, but whether each tensor in the weight tensorList is transposed must be consistent.
6) x does not support transposition.
7) Only ND in, ND out is supported.
8) Only non-quantized scenarios are supported.2 Single-single-single 1) Only splitItem 2/3 is supported.
2) Tensors in x and weight must be 2D with shapes (K, M) and (K, N) respectively. Tensors in y must be 3D with shape (g, M, N).
3) groupListOptional must be passed, and the last value must not exceed the first dimension of the tensor in x.
4) The first dimension of groupListOptional supports a maximum of 1024, that is, a maximum of 1024 groups.
5) x must be transposed and weight cannot be transposed.
6) Only ND in, ND out is supported.
7) Only non-quantized scenarios are supported.
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_grouped_matmul_v3.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate: resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor_New(const std::vector<int64_t>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate device memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from host to device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Compute strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call aclCreateTensor to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate device memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Copy data from the host to the device memory by calling aclrtMemcpy.
std::vector<T> hostData(size, 0);
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Create an aclTensor by calling the aclCreateTensor API.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
int CreateAclTensorList(const std::vector<std::vector<int64_t>>& shapes, void** deviceAddr,
aclDataType dataType, aclTensorList** tensor) {
int size = shapes.size();
std::vector<aclTensor*> tensors(size);
for (int i = 0; i < size; i++) {
int ret = CreateAclTensor<uint16_t>(shapes[i], deviceAddr + i, dataType, tensors.data() + i);
CHECK_RET(ret == ACL_SUCCESS, return ret);
}
*tensor = aclCreateTensorList(tensors.data(), size);
return ACL_SUCCESS;
}
int main() {
// 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
// Enter the deviceId based on your actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
// Handle the check based on your own needs.
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. Customize the construction based on the API interface.
std::vector<std::vector<int64_t>> xShape = {{512, 256}};
std::vector<std::vector<int64_t>> weightShape= {{2, 256, 256}};
std::vector<std::vector<int64_t>> biasShape = {{2, 256}};
std::vector<std::vector<int64_t>> yShape = {{512, 256}};
std::vector<int64_t> groupListShape = {{2}};
std::vector<int64_t> groupListData = {256, 512};
void* xDeviceAddr[1];
void* weightDeviceAddr[1];
void* biasDeviceAddr[1];
void* yDeviceAddr[1];
void* groupListDeviceAddr;
aclTensorList* x = nullptr;
aclTensorList* weight = nullptr;
aclTensorList* bias = nullptr;
aclTensor* groupedList = nullptr;
aclTensorList* scale = nullptr;
aclTensorList* offset = nullptr;
aclTensorList* antiquantScale = nullptr;
aclTensorList* antiquantOffset = nullptr;
aclTensorList* y = nullptr;
int64_t splitItem = 2;
int64_t groupType = 0;
// Create the x aclTensorList.
ret = CreateAclTensorList(xShape, xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the weight aclTensorList.
ret = CreateAclTensorList(weightShape, weightDeviceAddr, aclDataType::ACL_FLOAT16, &weight);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the bias aclTensorList.
ret = CreateAclTensorList(biasShape, biasDeviceAddr, aclDataType::ACL_FLOAT16, &bias);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create y aclTensorList.
ret = CreateAclTensorList(yShape, yDeviceAddr, aclDataType::ACL_FLOAT16, &y);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create group_list aclTensor.
ret = CreateAclTensor_New<int64_t>(groupListData, groupListShape, &groupListDeviceAddr, aclDataType::ACL_INT64, &groupedList);
CHECK_RET(ret == ACL_SUCCESS, return ret);
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// 3. Call the CANN operator library API.
// Call the first-phase API of aclnnGroupedMatmulV3.
ret = aclnnGroupedMatmulV3GetWorkspaceSize(x, weight, bias, scale, offset, antiquantScale, antiquantOffset, groupedList, splitItem, groupType, y, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmulGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize calculated by the first-phase API.
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnGroupedMatmulV3.
ret = aclnnGroupedMatmulV3(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnGroupedMatmul failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronously wait for task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output values and copy the results from the Device-side memory to the Host side. Modify based on the specific API definition.
for (int i = 0; i < 1; i++) {
auto size = GetShapeSize(yShape[i]);
std::vector<uint16_t> resultData(size, 0);
ret = aclrtMemcpy(resultData.data(), size * sizeof(resultData[0]), yDeviceAddr[i],
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t j = 0; j < size; j++) {
LOG_PRINT("result[%ld] is: %d\n", j, resultData[j]);
}
}
// 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
aclDestroyTensorList(x);
aclDestroyTensorList(weight);
aclDestroyTensorList(bias);
aclDestroyTensorList(y);
// 7. Release device resources. Modify based on the specific API definition.
for (int i = 0; i < 1; i++) {
aclrtFree(xDeviceAddr[i]);
aclrtFree(weightDeviceAddr[i]);
aclrtFree(biasDeviceAddr[i]);
aclrtFree(yDeviceAddr[i]);
}
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}