aclnnFastLayerNorm

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	×
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: Performs normalization calculation on the specified layer with a mean of 0 and a standard deviation of 1. Compared with the aclnnLayerNorm API, the aclnnFastLayerNorm API improves overall performance by 50%, with memory usage consistent with that of the GPU. Differences in precision exist due to the optimized accumulation order.
Formula:
$out = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + eps}} * weightOptional + biasOptional$ $meanOutOptional = \mathrm{E}[x]$ $rstdOutOptional = \frac{1}{ \sqrt{\mathrm{Var}[x] + eps}}$
Where E[x] represents the mean of the input, and Var[x] represents the variance of the input.

Function Prototype

Each operator is divided into a two-phase API. First call the "aclnnFastLayerNormGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that encapsulates the operator computation flow, and then call the "aclnnFastLayerNorm" API to perform the computation.

aclnnStatus aclnnFastLayerNormGetWorkspaceSize(
  const aclTensor   *input,
  const aclIntArray *normalizedShape,
  const aclTensor   *weightOptional,
  const aclTensor   *biasOptional,
  double             eps,
  aclTensor         *out,
  aclTensor         *meanOutOptional,
  aclTensor         *rstdOutOptional,
  uint64_t          *workspaceSize,
  aclOpExecutor    **executor)

aclnnStatus aclnnFastLayerNorm(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnFastLayerNormGetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
Input (aclTensor*)	Input	Represents the input for normalization calculation, corresponding to `x` in the formula.	Supports empty tensors. The shape is [A1,...,Ai,R1,...,Rj], where A1 through Ai represent dimensions that do not require normalization, and R1 through Rj represent dimensions that require normalization.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
normalizedShape（aclIntArray*）	Input	Indicates the dimensions that require normalization calculation.	The value is [R1,...,Rj], whose length is less than or equal to the length of the input shape. It cannot be empty. R1R2...*Rj is less than or equal to 583705600.	INT64	-	-	-
weightOptional (aclTensor*)	Input	Optional input parameter, representing the weight for normalization calculation. Corresponds to `weightOptional` in the formula.	Supports empty tensors. When `weightOptional` is non-empty: The data type is the same as that of the input `input` or is of FLOAT type, and when `biasOptional` exists, `weightOptional` has the same data type as `biasOptional`. The shape is equal to `normalizedShape`, which is [R1,...,Rj]. When `weightOptional` is empty, the API internally constructs a tensor with the shape [R1,...,Rj] and all elements set to 1. When `biasOptional` exists, `weightOptional` has the same data type as `biasOptional`. When `biasOptional` does not exist, `weightOptional` has the same data type as the input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
biasOptional (aclTensor*)	Input	An optional input parameter that specifies the offset for normalization calculation. Corresponds to `biasOptional` in the formula.	Supports empty tensors. When `biasOptional` is non-empty: The data type is the same as that of the input `input` or is of the FLOAT type, and when `weightOptional` exists, `biasOptional` has the same data type as `weightOptional`. The shape equals `normalizedShape`, which is [R1,...,Rj]. When `biasOptional` is empty, the API internally constructs a tensor with the shape [R1,...,Rj] and all elements set to 0. When `weightOptional` exists, `biasOptional` has the same data type as `weightOptional`. When `weightOptional` does not exist, `biasOptional` has the same data type as the input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
eps (double)	Input	Represents the value added to the denominator to ensure numerical stability. Corresponds to `eps` in the formula.	-	-	-	-	-
Output (aclTensor*).	Output	Represents the result of the normalization calculation. Corresponds to `out` in the formula.	Supports empty tensors. The data type is consistent with the data type of `input`. The shape must be equal to the shape of `input`, which is [A1,...,Ai,R1,...,Rj].	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
meanOutOptional (aclTensor*)	Output	Optional output, representing the mean after normalization. Corresponds to `meanOutOptional` in the formula.	Supports empty tensors. The data type is consistent with that of `input`. When `rstdOutOptional` exists, the shape is the same as that of `rstdOutOptional`, which is [A1,...,Ai,1,...,1] with j ones after Ai, matching the length of the axes to be normalized.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
rstdOutOptional (aclTensor*)	Output	Optional output, representing the reciprocal of the standard deviation after normalization. Corresponds to `rstdOutOptional` in the formula.	Supports an empty tensor. The data type is consistent with that of `input`. When `meanOutOptional` exists, the shape is the same as that of `meanOutOptional`, which is [A1,...,Ai,1,...,1], with j ones after Ai, keeping the same length as the axes to be normalized.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
workspaceSize (uint64_t*)	Output	Returns the workspace size to be applied for on the Device side.	-	-	-	-	-
executor (aclOpExecutor**)	Output	Returns the operator executor, which contains the operator computation flow.	-	-	-	-	-

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The input, normalizedShape, or out parameter is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The shape of input, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) exceeds 8 dimensions.
		The number of elements in normalizedShape exceeds 8.
		The data type of input, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) is not within the supported range.
		The dimension of normalizedShape is less than 1.
		weightOptional is not empty and its shape is not equal to normalizedShape.
		biasOptional is not empty and its shape is not equal to normalizedShape.
		The dimension of input is less than the dimension of normalizedShape.
		When the shape of input is right-aligned with normalizedShape, the corresponding dimension shapes are not equal.
		The shapes of input and out are inconsistent.
ACLNN_ERR_INNER_NULLPTR	561103	When the value of normalizedShape is [R1,...,Rj], the value of R1R2...*Rj exceeds 583705600.

aclnnFastLayerNorm

Parameters

Parameter	Input/Output	Description
Workspace.	Input	Address of the workspace memory applied for on the Device side.
workspaceSize	Input	Size of the workspace applied for on the Device side, obtained through the first-phase API aclnnFastLayerNormGetWorkspaceSize.
executor	Input	Operator executor that contains the operator computation process.
stream	Input	Specifies the stream for task execution.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

The shape of input, normalizedShape, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) does not exceed 8 dimensions.
Deterministic computation:
- aclnnFastLayerNorm defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_layer_norm.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensorMem(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
void aclCreateTensorP(const std::vector<T>& shape, void** deviceAddr, aclDataType dataType, aclTensor** tensor)
{
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
}

template <typename T>
void aclCreateIntArrayP(const std::vector<T>& hostData, aclIntArray** intArray)
{
    *intArray = aclCreateIntArray(hostData.data(), hostData.size());
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. Refer to the ACL API manual.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct the input and output. Customize the construction based on the API.
    std::vector<int64_t> xShape = {1, 2, 32};
    std::vector<int64_t> normShape = {32};
    std::vector<int64_t> meanShape = {1, 2, 1};
    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* meanDeviceAddr = nullptr;
    void* rstdDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclIntArray* norm = nullptr;
    aclTensor* weight = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    aclTensor* mean = nullptr;
    aclTensor* rstd = nullptr;
    std::vector<float> xHostData(64, 2.0);
    std::vector<int64_t> normData = {32};
    std::vector<float> weightHostData(32, 1.0);
    std::vector<float> biasHostData(32, 0.0);
    std::vector<float> outHostData(64, 0.0);
    std::vector<float> meanHostData(2, 0.0);
    std::vector<float> rstdHostData(2, 0.0);
    double eps = 1e-5;

    ret = CreateAclTensorMem(xHostData, xShape, &xDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(weightHostData, normShape, &weightDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(biasHostData, normShape, &biasDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(outHostData, xShape, &outDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(meanHostData, meanShape, &meanDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(rstdHostData, meanShape, &rstdDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    aclCreateTensorP(xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
    aclCreateIntArrayP(normData, &norm);
    aclCreateTensorP(normShape, &weightDeviceAddr, aclDataType::ACL_FLOAT, &weight);
    aclCreateTensorP(normShape, &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    aclCreateTensorP(xShape, &outDeviceAddr, aclDataType::ACL_FLOAT, &out);
    aclCreateTensorP(meanShape, &meanDeviceAddr, aclDataType::ACL_FLOAT, &mean);
    aclCreateTensorP(meanShape, &rstdDeviceAddr, aclDataType::ACL_FLOAT, &rstd);

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnFastLayerNorm.
    ret = aclnnFastLayerNormGetWorkspaceSize(x, norm, weight, bias, eps, out, mean, rstd, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFastLayerNormGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnFastLayerNorm.
    ret = aclnnFastLayerNorm(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnFastLayerNorm failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for the task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host side. Modify based on the specific API definition.
    auto size = GetShapeSize(xShape);
    std::vector<float> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy first result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("out result[%ld] is: %f\n", i, resultData[i]);
    }

    auto size1 = GetShapeSize(meanShape);
    std::vector<float> resultData1(size1, 0);
    ret = aclrtMemcpy(
        resultData1.data(), resultData1.size() * sizeof(resultData1[0]), meanDeviceAddr, size1 * sizeof(resultData1[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy second result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size1; i++) {
        LOG_PRINT("mean result[%ld] is: %f\n", i, resultData1[i]);
    }

    auto size2 = GetShapeSize(meanShape);
    std::vector<float> resultData2(size2, 0);
    ret = aclrtMemcpy(
        resultData2.data(), resultData2.size() * sizeof(resultData2[0]), rstdDeviceAddr, size2 * sizeof(resultData2[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy last result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size2; i++) {
        LOG_PRINT("rstd result[%ld] is: %f\n", i, resultData2[i]);
    }

    // 6. Release aclTensor and aclIntArray. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyIntArray(norm);
    aclDestroyTensor(weight);
    aclDestroyTensor(bias);
    aclDestroyTensor(out);
    aclDestroyTensor(mean);
    aclDestroyTensor(rstd);

    // 7. Release device resources.
    aclrtFree(xDeviceAddr);
    aclrtFree(weightDeviceAddr);
    aclrtFree(biasDeviceAddr);
    aclrtFree(outDeviceAddr);
    aclrtFree(meanDeviceAddr);
    aclrtFree(rstdDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}