aclnnAddRmsNormCast
Applicable Products
| Product | Supported |
|---|---|
| √ | |
| √ | |
| √ | |
| × | |
| × | |
| × |
Function
Description: The RmsNorm operator is a commonly used normalization operation in large models. The AddRmsNormCast operator fuses the Cast operator after AddRmsNorm to reduce data movement in and out.
Formula:
Function Prototype
Each operator uses a two-phase API. You must first call the aclnnAddRmsNormCastGetWorkspaceSize API to obtain the input parameters and calculate the required workspace size based on the computation process, and then call the aclnnAddRmsNormCast API to perform the computation.
aclnnStatus aclnnAddRmsNormCastGetWorkspaceSize(
const aclTensor *x1,
const aclTensor *x2,
const aclTensor *gamma,
double epsilon,
const aclTensor *y1Out,
const aclTensor *y2Out,
const aclTensor *rstdOut,
const aclTensor *xOut,
uint64_t *workspaceSize,
aclOpExecutor **executor)aclnnStatus aclnnAddRmsNormCast(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)aclnnAddRmsNormCastGetWorkspaceSize
Parameters
Parameter Input/Output Description Instruction Data Type Data Format Dimension (Shape) Non-contiguous Tensor x1 (aclTensor*) Input Represents the first input for the Add computation. Corresponds to `x1` in the formula. Supports null Tensor. BFLOAT16, FLOAT16 ND 1-8 × x2 (aclTensor*) Input Indicates the second input for the Add computation. Corresponding to `x2` in the formula. - Supports null Tensor.
- The shape and data type need to remain consistent with the shape and data type of `x1`.
FLOAT16, BFLOAT16 ND 1-8 × gamma (aclTensor*) Input Indicates the scaling factor (weight) for RMS normalization. Corresponds to `gamma` in the formula. - Supports null tensor.
- The data type remains consistent with that of `x1`.
- The shape needs to remain consistent with the last several dimensions of `x1`, which are the dimensions to be normalized for `x1`.
FLOAT16, BFLOAT16 ND 1-8 × epsilon (double) Input Value added to the denominator to ensure numerical stability. Corresponds to `epsilon` in the formula. Recommended value: 1e-6. - - - - y1Out (aclTensor*) Output Represents the output data after normalization and type conversion. Corresponds to `y1Out` in the formula. - Supports null Tensor.
- The shape and data format must remain consistent with the input `x1`.
FLOAT32 ND 1-8 × y2Out (aclTensor*) Output Indicates the normalized output data, corresponding to `y2Out` in the formula. - Supports null tensor.
- The shape, data format, and data type must remain consistent with the input `x1`.
FLOAT16, BFLOAT16 ND 1-8 × rstdOut（aclTensor*） Output Indicates the reciprocal of the standard deviation after normalization. Corresponds to the reciprocal of `Rms(x)` in the formula. - Supports null Tensor.
- Must remain consistent with the data format of the input `x1`.
- The shape must remain consistent with the first several dimensions of the input `x1`'s shape, where the first several dimensions refer to the dimensions of `x1` minus the dimensions of `gamma`, indicating the dimensions that do not require normalization.
- When the input `x1` is a null tensor, `rstdOut` must also be a null tensor.
FLOAT32 ND 1-8 × xOut (aclTensor*) Output Represents the result of the Add operation. Corresponds to `x` in the formula. - Supports null Tensor.
- The shape, data format, and data type must remain consistent with the input `x1`.
FLOAT16, BFLOAT16 ND 1-8 × workspaceSize (uint64_t*) Output Returns the workspace size applied on the Device side. - - - - - executor（aclOpExecutor**） Output Returns the op executor, which contains the operator computation flow. - - - - - Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Return Code Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 The input parameter is a required input, output, or required attribute, and is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type of the input or output is not within the supported range. ACLNN_ERR_INNER_TILING_ERROR 561002 The input and output do not meet the requirements in the parameter description above.
aclnnAddRmsNormCast
Parameters
Parameter Input/Output Description workspace Input Workspace memory address applied for on the Device side. workspaceSize Input Workspace size applied for on the Device side, obtained from the first-phase API aclnnAddRmsNormCastGetWorkspaceSize. executor Input The operator executor that contains the operator computation process. stream Input Specifies the stream for task execution. Return Value
aclnnStatus: return code. For details, see aclnn Return Code.
Constraints
- Dimension boundary description:
The size of each dimension in the shape of x1, x2, gamma, y1Out, y2Out, rstdOut, and xOut is not greater than the maximum INT32 value of 2147483647.
Boundary value scenario description:
When the input is Inf, the output is Inf.
- When the input is NaN, the output is NaN.
Deterministic computation:
- aclnnAddRmsNormCast defaults to a deterministic implementation.
Example
The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_add_rms_norm_cast.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
int64_t shape_size = 1;
for (auto i : shape) {
shape_size *= i;
}
return shape_size;
}
int Init(int32_t deviceId, aclrtStream* stream)
{
// Boilerplate: resource initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(
const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
aclTensor** tensor)
{
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to apply for device-side memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Copy data from the host to the device memory by calling aclrtMemcpy.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Compute the strides of a contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Create an aclTensor by calling the aclCreateTensor API.
*tensor = aclCreateTensor(
shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
*deviceAddr);
return 0;
}
int main()
{
// 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
// Set deviceId based on the actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
// Handle the check based on your own needs.
CHECK_RET(ret == 0, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the inputs and outputs. Customize the construction based on the API.
std::vector<int64_t> xShape = {2, 16};
std::vector<int64_t> gammaShape = {16};
std::vector<int64_t> yShape = {2, 16};
std::vector<int64_t> rstdShape = {2, 1};
void* x1DeviceAddr = nullptr;
void* x2DeviceAddr = nullptr;
void* gammaDeviceAddr = nullptr;
void* y1DeviceAddr = nullptr;
void* y2DeviceAddr = nullptr;
void* rstdDeviceAddr = nullptr;
void* xDeviceAddr = nullptr;
aclTensor* x1 = nullptr;
aclTensor* x2 = nullptr;
aclTensor* gamma = nullptr;
aclTensor* y1 = nullptr;
aclTensor* y2 = nullptr;
aclTensor* rstd = nullptr;
aclTensor* x = nullptr;
std::vector<short> x1HostData = {0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700};
std::vector<short> x2HostData = {0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700};
std::vector<short> gammaHostData = {0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700};
std::vector<float> y1HostData = {0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7,
0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7};
std::vector<short> y2HostData = {0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700};
std::vector<float> rstdHostData = {1, 2};
std::vector<short> xHostData = {0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700,
0x0000, 0x3C00, 0x4000, 0x4200, 0x4400, 0x4500, 0x4600, 0x4700};
float epsilon = 1e-6;
// Create the x1 aclTensor.
ret = CreateAclTensor(x1HostData, xShape, &x1DeviceAddr, aclDataType::ACL_FLOAT16, &x1);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x2 aclTensor.
ret = CreateAclTensor(x2HostData, xShape, &x2DeviceAddr, aclDataType::ACL_FLOAT16, &x2);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the gamma aclTensor.
ret = CreateAclTensor(gammaHostData, gammaShape, &gammaDeviceAddr, aclDataType::ACL_FLOAT16, &gamma);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the y1 aclTensor.
ret = CreateAclTensor(y1HostData, yShape, &y1DeviceAddr, aclDataType::ACL_FLOAT, &y1);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the y2 aclTensor.
ret = CreateAclTensor(y2HostData, yShape, &y2DeviceAddr, aclDataType::ACL_FLOAT16, &y2);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the rstd aclTensor.
ret = CreateAclTensor(rstdHostData, rstdShape, &rstdDeviceAddr, aclDataType::ACL_FLOAT, &rstd);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the x aclTensor.
ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// 3. Call the CANN operator library API. Replace with the specific API.
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// Call the first-phase API of aclnnAddRmsNormCast.
ret = aclnnAddRmsNormCastGetWorkspaceSize(x1, x2, gamma, epsilon, y1, y2, rstd, x, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddRmsNormCastGetWorkspaceSize failed. ERROR: %d\n", ret);
return ret);
// Apply for device memory based on the workspaceSize computed by the first-phase API.
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
}
// Call the second-phase API of aclnnAddRmsNormCast.
ret = aclnnAddRmsNormCast(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddRmsNormCast failed. ERROR: %d\n", ret); return ret);
// 4. (Boilerplate) Wait synchronously for the task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value, and copy the result from the device memory to the host memory. Modify based on the specific API definition.
auto size = GetShapeSize(yShape);
std::vector<float> resultData(size, 0);
ret = aclrtMemcpy(
resultData.data(), resultData.size() * sizeof(resultData[0]), y1DeviceAddr, size * sizeof(resultData[0]),
ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("y1 result[%ld] is: %f\n", i, resultData[i]);
}
std::vector<int8_t> resultData1(size, 0);
ret = aclrtMemcpy(
resultData1.data(), resultData1.size() * sizeof(resultData1[0]), y2DeviceAddr, size * sizeof(resultData1[0]),
ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("y2 result[%ld] is: %d\n", i, resultData1[i]);
}
// 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
aclDestroyTensor(x1);
aclDestroyTensor(x2);
aclDestroyTensor(gamma);
aclDestroyTensor(y1);
aclDestroyTensor(y2);
aclDestroyTensor(rstd);
aclDestroyTensor(x);
// 7. Release device resources. Modify based on the specific API definition.
aclrtFree(x1DeviceAddr);
aclrtFree(x2DeviceAddr);
aclrtFree(xDeviceAddr);
aclrtFree(gammaDeviceAddr);
aclrtFree(y2DeviceAddr);
aclrtFree(y1DeviceAddr);
aclrtFree(rstdDeviceAddr);
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}