aclnnAddmmWeightNz
Applicable Products
| Product | Supported |
|---|---|
| √ | |
| √ | |
| √ | |
| × | |
| × | |
| × |
Function
Description: Computes α multiplied by the product of mat1 and mat2, then adds the product of β and self. Compared with the original addmm API, the new API supports the NZ format for mat2.
Formula:
Example:
For the aclnnAddmmWeightNz API, the shape of self is [n,], the shape of mat1 is [m, k], the shape of mat2 is [k, n], the shape of the Matrix Multiplication result of mat1 and mat2 is [m, n], and the shape of self can be broadcast to [m, n].
For the aclnnAddmmWeightNz API, the shape of self is [1, n], the shape of mat1 is [m, k], the shape of mat2 is [k, n], the shape of the matrix multiplication result of mat1 and mat2 is [m, n], and the shape of self can be broadcast to [m, n].
For the aclnnAddmmWeightNz API, the shape of self is [m, n], the shape of mat1 is [m, k], the shape of mat2 is [k, n], and the shape of the matrix multiplication result of mat1 and mat2 is [m, n].
Function Prototype
Each operator uses a two-phase API. You must first call the "aclnnAddmmWeightNzGetWorkspaceSize" API to obtain the input parameters and calculate the required workspace size based on the computation process, and then call the "aclnnAddmmWeightNz" API to perform the computation.
aclnnStatus aclnnAddmmWeightNzGetWorkspaceSize(
const aclTensor *self,
const aclTensor *mat1,
const aclTensor *mat2,
const aclScalar *beta,
const aclScalar *alpha,
aclTensor *out,
int8_t cubeMathType,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnAddmmWeightNz(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnAddmmWeightNzGetWorkspaceSize
Parameters
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-Contiguous Tensor self Input Represents the bias matrix, which is self in the formula. - The data type must satisfy the data type deduction rules with mat1@mat2 (see Deduction Relationship and Constraints).
- Must satisfy the broadcast relationship with mat1@mat2.
- self supports shapes of (n), (1, n), and (m, n).
BFLOAT16, FLOAT16, FLOAT32 ND 1-2 √ mat1 Input Represents the first matrix of the matrix multiplication, mat1 in the formula. - The data type must satisfy the data type deduction rules with mat2 (see deduction relationship and Constraints).
- Must satisfy the broadcast relationship with self and mat2.
- When mat1 is not transposed, the dimension representation is: (m, k).
- When mat1 is transposed, the dimension representation is: (k, m).
BFLOAT16, FLOAT16 ND 2 √ mat2 Input Represents the second matrix in the matrix multiplication, mat2 in the formula. - The data type must satisfy the data type deduction rules with mat1 (see Deduction Relationship and Constraints).
- The reduce dimension of mat2 must be equal in size to the reduce dimension of mat1.
- Must satisfy the broadcast relationship with self and mat1.
- When the mat2 matrix is not transposed, the NZ format dimensions are represented as: (n1, k1, k0, n0), where k0 = 16 and n0 = 16. The k in mat1 shape and k1 in mat2 shape must satisfy the following relationship: ceil(k, k0) = k1. The n1 in mat2 shape and n in out must satisfy the following relationship: ceil(n, n0) = n1.
- When the mat2 matrix is transposed, the NZ format dimensions are represented as: (k1, n1, n0, k0), where n0 = 16 and k0 = 16. The k in mat1 shape and k1 in mat2 shape must satisfy the following relationship: ceil(k, k0) = k1. The n1 in mat2 shape and n in out must satisfy the following relationship: ceil(n, n0) = n1.
BFLOAT16, FLOAT16 NZ 4 - beta(β) Input Represents β in the formula. - FLOAT, FLOAT16, DOUBLE, INT8, INT16, INT32, INT64, UINT8 - - - alpha(α) Input Represents α in the formula. - FLOAT, FLOAT16, DOUBLE, INT8, INT16, INT32, INT64, UINT8 - - - out Output Represents the output matrix of matrix multiplication, out in the formula. The data type must be consistent with the data type derived from self and mat2 (see Deduction Relationship and Constraints). BFLOAT16, FLOAT16, FLOAT32 ND 2 - cubeMathType Input Specifies the computation logic of the Cube unit. If the input data types have a deduction relationship, this parameter processes the data type after deduction by default. The supported enumeration values are as follows: - 0: KEEP_DTYPE. Keeps the input data type for computation.
- 1: ALLOW_FP32_DOWN_PRECISION. Allows down-precision computation of the input data.
- 2: USE_FP16. Allows down-precision of the input to FLOAT16 for computation.
- 3: USE_HF32. Allows down-precision of the input to the HFLOAT32 data type for computation.
INT8 - - - workspaceSize Output Returns the workspace size to be applied on the Device side. - - - - - executor Output Returns the operator executor, which contains the operator computation process. - - - - - Atlas A2 training products/Atlas A2 inference products ,Atlas A3 training products/Atlas A3 inference products ,Ascend 950PR/Ascend 950DT :cubeMathType=1: When the input data type is FLOAT32, it will be converted to HFLOAT32 for computation. When the input is of other data types, no processing is performed.
cubeMathType=2: This option is not supported when the input data type is BFLOAT16.
cubeMathType=3: When the input data type is FLOAT32, it will be converted to HFLOAT32 for computation. This option is not supported when the input is of other data types.
Atlas training products ,Atlas inference products :BFLOAT16 data type is not supported;
When the input data type is FLOAT32, cubeMathType=0 is not supported;
When cubeMathType=1 and the input data type is FLOAT32, it will be converted to FLOAT16 for computation. When the input is of other data types, no processing is performed;
cubeMathType=3 is not supported.
Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Value Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 The passed self, mat1, mat2, or out is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type and data format of self and mat2 are not within the supported range. mat1 and mat2 do not satisfy the multiplication condition. The shapes of out and mat1@mat2 are inconsistent.
aclnnAddmmWeightNz
Parameters
Parameter Input/Output Description workspace Input Memory address of the workspace applied on the Device side. workspaceSize Input Size of the workspace applied on the Device side, obtained through the first-phase API aclnnAddmmWeightNzGetWorkspaceSize. executor Input Operator executor that contains the operator computation flow. stream Input Specifies the stream for task execution. Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
Deterministic computation description: aclnnAddmmWeightNz defaults to a deterministic implementation.
Computation consistency description
Atlas training products ,Atlas inference products :When the strong consistency computation feature is enabled, the computation result is deterministic, and multiple executions produce the same output. In addition, the computation result is independent of the data location.
aclnnAddmmWeightNz defaults to a non-deterministic implementation and supports enabling deterministic computation via aclrtCtxSetSysParamOpt.
For example, during Matrix Multiplication, the accumulation order of different basic blocks may vary, which can cause slight differences in the computation results of the same data across different rows. However, when deterministic computation is enabled, the computation results will be identical as long as the inputs are the same, even across different rows.
Data type deduction is not supported when one of the two inputs, mat1 and mat2, is BFLOAT16 and the other is FLOAT or FLOAT16.
It is not supported when one of the last two axes of mat2 is 1, i.e., k=1 or n=1.
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include <cmath>
#include "acl/acl.h"
#include "aclnnop/aclnn_addmm.h"
#include "aclnnop/aclnn_trans_matmul_weight.h"
#include "aclnnop/aclnn_cast.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
// Convert the uint16_t representation of FP16 to float representation.
float Fp16ToFloat(uint16_t h) {
int s = (h >> 15) & 0x1; // sign
int e = (h >> 10) & 0x1F; // exponent
int f = h & 0x3FF; // fraction
if (e == 0) {
// Zero or Denormal
if (f == 0) {
return s ? -0.0f : 0.0f;
}
// Denormals
float sig = f / 1024.0f;
float result = sig * pow(2, -24);
return s ? -result : result;
} else if (e == 31) {
// Infinity or NaN
return f == 0 ? (s ? -INFINITY : INFINITY) : NAN;
}
// Normalized
float result = (1.0f + f / 1024.0f) * pow(2, e - 15);
return s ? -result : result;
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate for resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to apply for memory on the device side.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from the host side to the device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Compute the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
template <typename T>
int CreateAclTensorWeight(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = static_cast<uint64_t>(GetShapeSize(shape));
const aclIntArray* mat2Size = aclCreateIntArray(shape.data(), shape.size());
auto ret = aclnnCalculateMatmulWeightSize(mat2Size, &size);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnCalculateMatmulWeightSize failed. ERROR: %d\n", ret); return ret);
size *= sizeof(T);
// Call aclrtMalloc to apply for memory on the device side.
ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from the host side to the device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Compute the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
std::vector<int64_t> storageShape;
storageShape.push_back(GetShapeSize(shape));
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
storageShape.data(), storageShape.size(), *deviceAddr);
return 0;
}
int main() {
// 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
// Fill in the deviceId based on the actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. Customize the construction based on the API.
std::vector<int64_t> selfShape = {16};
std::vector<int64_t> mat1Shape = {16, 32};
std::vector<int64_t> mat2Shape = {32, 16};
std::vector<int64_t> outShape = {16, 16};
void* selfDeviceAddr = nullptr;
void* mat1DeviceAddr = nullptr;
void* mat2DeviceAddr = nullptr;
void* outDeviceAddr = nullptr;
aclTensor* self = nullptr;
aclTensor* mat1 = nullptr;
aclTensor* mat2 = nullptr;
aclTensor* out = nullptr;
aclScalar* alpha = nullptr;
aclScalar* beta = nullptr;
std::vector<uint16_t> selfHostData(16, 0x3C00); // float16_t uses 0x3C00 to represent int_16 1.
std::vector<uint16_t> mat1HostData(512, 0x3C00); // float16_t uses 0x3C00 to represent int_16 1.
std::vector<uint16_t> mat2HostData(512, 0x3C00); // float16_t uses 0x3C00 to represent int_16 1.
std::vector<uint16_t> outHostData(256, 0);
float alphaValue = 1.0f;
float betaValue = 1.0f;
// Create the self aclTensor.
ret = CreateAclTensor(selfHostData, selfShape, &selfDeviceAddr, aclDataType::ACL_FLOAT16, &self);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the mat1 aclTensor.
ret = CreateAclTensor(mat1HostData, mat1Shape, &mat1DeviceAddr, aclDataType::ACL_FLOAT16, &mat1);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the mat2 aclTensor.
ret = CreateAclTensorWeight(mat2HostData, mat2Shape, &mat2DeviceAddr, aclDataType::ACL_FLOAT16, &mat2);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the out aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the alpha aclScalar.
alpha = aclCreateScalar(&alphaValue,aclDataType::ACL_FLOAT);
CHECK_RET(alpha != nullptr, return ret);
// Create beta aclScalar.
beta = aclCreateScalar(&betaValue,aclDataType::ACL_FLOAT);
CHECK_RET(beta != nullptr, return ret);
// 3. Call the CANN operator library API. Replace with the specific API name.
int8_t cubeMathType = 1;
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// Call TransWeight.
ret = aclnnTransMatmulWeightGetWorkspaceSize(mat2, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeightGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize computed by the first-phase API.
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnTransMatmulWeight.
ret = aclnnTransMatmulWeight(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnTransMatmulWeight failed. ERROR: %d\n", ret); return ret);
// Call the first-phase API of aclnnAddmmWeightNz.
uint64_t workspaceSizeMm = 0;
ret = aclnnAddmmWeightNzGetWorkspaceSize(self, mat1, mat2, beta, alpha, out, cubeMathType, &workspaceSizeMm, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddmmWeightNzGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Apply for device memory based on the workspaceSize computed by the first-phase API.
void* workspaceAddrMm = nullptr;
if (workspaceSizeMm > 0) {
ret = aclrtMalloc(&workspaceAddrMm, workspaceSizeMm, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnAddmmWeightNz.
ret = aclnnAddmmWeightNz(workspaceAddrMm, workspaceSizeMm, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddmmWeightNz failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Synchronously wait for the task to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device memory to the host memory. Modify this based on the specific API definition.
auto size = GetShapeSize(outShape);
std::vector<uint16_t> resultData(size, 0);
ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr,
size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
// fp16 data cannot be directly printed in C. Read it as uint16 and manually convert it to float-represented fp16 through binary conversion.
for (int64_t i = 0; i < size; i++) {
float fp16Float = Fp16ToFloat(resultData[i]);
LOG_PRINT("result[%ld] is: %f\n", i, fp16Float);
}
// 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
aclDestroyTensor(self);
aclDestroyTensor(mat1);
aclDestroyTensor(mat2);
aclDestroyScalar(beta);
aclDestroyScalar(alpha);
aclDestroyTensor(out);
// 7. Release device resources. Modify based on the specific API definition.
aclrtFree(selfDeviceAddr);
aclrtFree(mat1DeviceAddr);
aclrtFree(mat2DeviceAddr);
aclrtFree(outDeviceAddr);
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
if (workspaceSizeMm > 0) {
aclrtFree(workspaceAddrMm);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}