aclnnLayerNorm&aclnnLayerNormWithImplMode

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	√
Atlas inference products	×
Atlas training products	√

Function

Description: Performs normalization calculation with a mean of 0 and a standard deviation of 1 on the specified layer. Compared with the aclnnLayerNorm API, the aclnnLayerNormWithImplMode API allows you to select different normalization implementation strategies by configuring the impl_mode parameter, to adapt to different app scenarios and performance requirements.
Formula:
$out = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + eps}} * weightOptional + biasOptional$ $meanOutOptional = \mathrm{E}[x]$ $rstdOutOptional = \frac{1}{ \sqrt{\mathrm{Var}[x] + eps}}$
Where E[x] represents the mean of the input, and Var[x] represents the variance of the input.

Function Prototype

Each operator is divided into a two-phase API. You must first call the "aclnnLayerNormGetWorkspaceSize" or "aclnnLayerNormWithImplModeGetWorkspaceSize" API to obtain the workspace size required for computation and the executor that includes the operator computation process, and then call the "aclnnLayerNorm" or "aclnnLayerNormWithImplMode" API to perform the computation.

aclnnStatus aclnnLayerNormGetWorkspaceSize(
  const aclTensor   *input,
  const aclIntArray *normalizedShape,
  const aclTensor   *weightOptional,
  const aclTensor   *biasOptional,
  double             eps,
  aclTensor         *out,
  aclTensor         *meanOutOptional,
  aclTensor         *rstdOutOptional,
  uint64_t          *workspaceSize,
  aclOpExecutor    **executor)

aclnnStatus aclnnLayerNorm(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnStatus aclnnLayerNormWithImplModeGetWorkspaceSize(
  const aclTensor    *input,
  const aclIntArray  *normalizedShape,
  const aclTensor    *weightOptional,
  const aclTensor    *biasOptional,
  double              eps,
  aclTensor          *out,
  aclTensor          *meanOutOptional,
  aclTensor          *rstdOutOptional,
  int32_t             implMode,
  uint64_t           *workspaceSize,
  aclOpExecutor     **executor)

aclnnStatus aclnnLayerNormWithImplMode(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnLayerNormGetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (shape)	Non-contiguous Tensor
input (aclTensor*)	Input	Indicates the input for normalization calculation, corresponding to `x` in the formula.	Supports null Tensor. The shape is [A1,...,Ai,R1,...,Rj], where A1 through Ai represent dimensions that do not require normalization, and R1 through Rj represent dimensions that require normalization.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
normalizedShape (aclIntArray*)	Input	Indicates the dimension that requires normalization calculation.	The value is [R1,...,Rj], with a length less than or equal to the shape length of the input. Null is not supported.	INT64	-	-	-
weightOptional (aclTensor*)	Input	Optional input parameter, indicating the weight for normalization calculation. Corresponds to `weightOptional` in the formula.	Supports null Tensor. When `weightOptional` is non-null: The data type is consistent with the input `input` or is FLOAT type, and when `biasOptional` is present, `weightOptional` has the same data type as `biasOptional`. The shape is equal to `normalizedShape`, which is [R1,...,Rj]. When `weightOptional` is null, the internal API constructs a Tensor with shape [R1,...,Rj] and all data set to 1. When `biasOptional` is present, `weightOptional` has the same data type as `biasOptional`. When `biasOptional` is absent, `weightOptional` has the same data type as the input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
biasOptional (aclTensor*)	Input	Optional input parameter. Indicates the offset for normalization calculation. Corresponds to `biasOptional` in the formula.	Supports null Tensor. When `biasOptional` is non-null: The data type is consistent with the input `input` or is of FLOAT type, and when `weightOptional` is present, `biasOptional` has the same data type as `weightOptional`. The shape is equal to `normalizedShape`, which is [R1,...,Rj]. When `biasOptional` is null, the internal interface will construct a Tensor with shape [R1,...,Rj] and all data being 0. When `weightOptional` is present, `biasOptional` has the same data type as `weightOptional`. When `weightOptional` is absent, `biasOptional` has the same data type as the input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
eps (double)	Input	Indicates the value added to the denominator to ensure numerical stability. Corresponds to `eps` in the formula.	-	-	-	-	-
out (aclTensor*)	Output	Indicates the result of the normalization calculation. Corresponds to `out` in the formula.	Supports null Tensor. The data type is consistent with the data type of `input`. The shape must be equal to the shape of `input`, i.e., [A1,...,Ai,R1,...,Rj].	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
meanOutOptional (aclTensor*)	Output	Optional output. Indicates the mean after normalization. Corresponds to `meanOutOptional` in the formula.	Supports null Tensor. The data type is consistent with that of `input`. When `rstdOutOptional` is present, the shape is the same as that of `rstdOutOptional`, in the format [A1,...,Ai,1,...,1], where there are j ones after Ai, and the axis length matches that of the dimension requiring normalization.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
rstdOutOptional (aclTensor*)	Output	Optional output. Indicates the reciprocal of the standard deviation after normalization. Corresponds to `rstdOutOptional` in the formula.	Supports null Tensor. The data type is consistent with the data type of `input`. When `meanOutOptional` is present, the shape is the same as that of `meanOutOptional`, which is [A1,...,Ai,1,...,1], with j ones after Ai, keeping the same axis length as the dimension to be normalized.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
workspaceSize (uint64_t*)	Output	Returns the workspace size to be applied for on the Device side.	-	-	-	-	-
executor (aclOpExecutor**)	Output	Returns the op executor, which includes the operator computation flow.	-	-	-	-	-

Atlas training products, Atlas 200I/500 A2 inference products: The data type of the parameters input, weightOptional, biasOptional, out, meanOutOptional, and rstdOutOptional does not support BFLOAT16.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The input, normalizedShape, or out is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The dimension of the shape of input, normalizedShape, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) exceeds 8.
		The data type of input, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) is not within the supported range.
		The normalizedShape dimension is less than 1.
		weightOptional is non-null and its shape is not equal to normalizedShape.
		biasOptional is non-null and its shape is not equal to normalizedShape.
		The input dimension is less than the normalizedShape dimension.
		The input shape is not equal to the corresponding dimension shape when right-aligned with normalizedShape.
		The shapes of input and out are inconsistent.

aclnnLayerNorm

Parameters

Parameter	Input/Output	Description
workspace	Input	Address of the workspace memory applied on the Device side.
workspaceSize	Input	Size of the workspace applied on the Device side, obtained through the first-phase API aclnnLayerNormGetWorkspaceSize.
Executor	Input	Operator executor that includes the calculation process.
Stream	Input	Specifies the stream for task execution.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

aclnnLayerNormWithImplModeGetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (shape)	Non-contiguous Tensor
input (aclTensor*)	Input	Indicates the input for normalization calculation, corresponding to `x` in the formula.	Supports null Tensor. The shape is [A1,...,Ai,R1,...,Rj], where A1 through Ai represent dimensions that do not require normalization, and R1 through Rj represent dimensions that require normalization.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
normalizedShape (aclIntArray*)	Input	Indicates the dimensions that require norm calculation.	The value is [R1,...,Rj], with a length less than or equal to the shape length of the input input. Null is not supported.	INT64	-	-	-
weightOptional (aclTensor*)	Input	Optional input parameter, indicating the weight for normalization calculation. Corresponds to `weightOptional` in the formula.	Supports null Tensor. When `weightOptional` is non-null: The data type is consistent with input `input` or is FLOAT32, and when `biasOptional` is present, `weightOptional` has the same data type as `biasOptional`. The shape is equal to `normalizedShape`, which is [R1,...,Rj]. When `weightOptional` is null, the internal interface will construct a Tensor with shape [R1,...,Rj] and all data being 1. When `biasOptional` is present, `weightOptional` has the same data type as `biasOptional`. When `biasOptional` is absent, `weightOptional` has the same data type as input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
biasOptional (aclTensor*)	Input	Optional input parameter. Indicates the offset for normalization calculation. Corresponds to `biasOptional` in the formula.	Supports null Tensor. When `biasOptional` is non-null: The data type is consistent with the input `input` or is FLOAT32, and when `weightOptional` is present, `biasOptional` has the same data type as `weightOptional`. The shape is equal to `normalizedShape`, which is [R1,...,Rj]. When `biasOptional` is null, the internal interface constructs a Tensor with shape [R1,...,Rj] and all data being 0. When `weightOptional` is present, `biasOptional` has the same data type as `weightOptional`. When `weightOptional` is absent, `biasOptional` has the same data type as the input `input`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
eps (double)	Input	Indicates the value added to the denominator to ensure numerical stability. Corresponds to `eps` in the formula.	-	-	-	-	-
out (aclTensor*)	Output	Indicates the result of the normalization calculation. Corresponds to `out` in the formula.	Supports null Tensor. The data type is consistent with the data type of `input`. The shape must be equal to the shape of `input`, which is [A1,...,Ai,R1,...,Rj].	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
meanOutOptional (aclTensor*)	Output	Optional output, representing the mean after normalization. Corresponds to `meanOutOptional` in the formula.	Supports null Tensor. The data type is consistent with that of `input`. The shape is [A1,...,Ai,1,...,1], where there are j ones after Ai, and the axis length matches that of the dimension to be normalized.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
rstdOutOptional (aclTensor*)	Output	Optional output. Indicates the reciprocal of the standard deviation after normalization. Corresponds to `rstdOutOptional` in the formula.	Supports null Tensor. The data type is consistent with that of `input`. The shape is [A1,...,Ai,1,...,1], with a total of j ones after Ai, keeping the same axis length as the dimension to be normalized.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
implMode (int32_t)	Input	Specifies the computation mode for kernel selection.	Supports configuration values of 0, 1, and 2, with a default of 0. 0 indicates high-precision mode, 1 indicates high-performance mode, and 2 indicates FLOAT16 computation mode. Use high-performance mode with caution, as it may cause precision loss. The FLOAT16 computation mode is supported only when all inputs are of the FLOAT16 type and offers the lowest computation precision.	INT32	-	-	-
workspaceSize (uint64_t*)	Output	Returns the workspace size to be applied for on the Device side.	-	-	-	-	-
Executor (aclOpExecutor**)	Output	Returns the operator executor, including the operator computation flow.	-	-	-	-	-

Atlas training products, Atlas 200I/500 A2 inference products: The data type of the parameters input, weightOptional, biasOptional, out, meanOutOptional, and rstdOutOptional does not support BFLOAT16.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	Indicates that the input, normalizedShape, or out is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The shape of input, normalizedShape, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) exceeds 8 dimensions.
		The data type of input, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) is not within the supported range.
		The dimension of normalizedShape is less than 1.
		weightOptional is non-null and its shape is not equal to normalizedShape.
		biasOptional is non-null and its shape is not equal to normalizedShape.
		The dimension of input is less than the dimension of normalizedShape.
		When the shape of input is right-aligned with normalizedShape, the corresponding dimension shapes are not equal.
		The value of implMode is not within the range of 0, 1, and 2.
		The value of implMode is 2 and the input data types are not all FLOAT16.
		The shapes of input and out are inconsistent.

aclnnLayerNormWithImplMode

Parameters

Parameter	Input/Output	Description
workspace	Input	Memory address of the workspace applied for on the Device side.
workspaceSize	Input	Size of the workspace applied for on the Device side, obtained from the first-phase API aclnnLayerNormWithImplModeGetWorkspaceSize.
Executor	Input	Operator executor that includes the calculation process.
stream	Input	Specifies the Stream for task execution.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

The shape of input, normalizedShape, weightOptional (when non-null), biasOptional (when non-null), out, meanOutOptional (when non-null), or rstdOutOptional (when non-null) does not exceed 8 dimensions.
Deterministic computation:
- aclnnLayerNorm defaults to a deterministic implementation.
- aclnnLayerNormWithImplMode defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

aclnnLayerNorm example:

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_layer_norm.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensorMem(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
void aclCreateTensorP(const std::vector<T>& shape, void** deviceAddr, aclDataType dataType, aclTensor** tensor)
{
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
}

template <typename T>
void aclCreateIntArrayP(const std::vector<T>& hostData, aclIntArray** intArray)
{
    *intArray = aclCreateIntArray(hostData.data(), hostData.size());
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Custom construction is required based on the API.
    std::vector<int64_t> xShape = {1, 2, 32};
    std::vector<int64_t> normShape = {32};
    std::vector<int64_t> meanShape = {1, 2, 1};
    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* meanDeviceAddr = nullptr;
    void* rstdDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclIntArray* norm = nullptr;
    aclTensor* weight = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    aclTensor* mean = nullptr;
    aclTensor* rstd = nullptr;
    std::vector<float> xHostData(64, 2.0);
    std::vector<int64_t> normData = {32};
    std::vector<float> weightHostData(32, 1.0);
    std::vector<float> biasHostData(32, 0.0);
    std::vector<float> outHostData(64, 0.0);
    std::vector<float> meanHostData(2, 0.0);
    std::vector<float> rstdHostData(2, 0.0);
    double eps = 1e-5;

    ret = CreateAclTensorMem(xHostData, xShape, &xDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(weightHostData, normShape, &weightDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(biasHostData, normShape, &biasDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(outHostData, xShape, &outDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(meanHostData, meanShape, &meanDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    ret = CreateAclTensorMem(rstdHostData, meanShape, &rstdDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    aclCreateTensorP(xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
    aclCreateIntArrayP(normData, &norm);
    aclCreateTensorP(normShape, &weightDeviceAddr, aclDataType::ACL_FLOAT, &weight);
    aclCreateTensorP(normShape, &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    aclCreateTensorP(xShape, &outDeviceAddr, aclDataType::ACL_FLOAT, &out);
    aclCreateTensorP(meanShape, &meanDeviceAddr, aclDataType::ACL_FLOAT, &mean);
    aclCreateTensorP(meanShape, &rstdDeviceAddr, aclDataType::ACL_FLOAT, &rstd);

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnLayerNorm.
    ret = aclnnLayerNormGetWorkspaceSize(x, norm, weight, bias, eps, out, mean, rstd, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnLayerNormGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnLayerNorm.
    ret = aclnnLayerNorm(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnLayerNorm failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host. Modify based on the specific API definition.
    auto size = GetShapeSize(xShape);
    std::vector<float> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy first result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("out result[%ld] is: %f\n", i, resultData[i]);
    }

    auto size1 = GetShapeSize(meanShape);
    std::vector<float> resultData1(size1, 0);
    ret = aclrtMemcpy(
        resultData1.data(), resultData1.size() * sizeof(resultData1[0]), meanDeviceAddr, size1 * sizeof(resultData1[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy second result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size1; i++) {
        LOG_PRINT("mean result[%ld] is: %f\n", i, resultData1[i]);
    }

    auto size2 = GetShapeSize(meanShape);
    std::vector<float> resultData2(size2, 0);
    ret = aclrtMemcpy(
        resultData2.data(), resultData2.size() * sizeof(resultData2[0]), rstdDeviceAddr, size2 * sizeof(resultData2[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy last result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size2; i++) {
        LOG_PRINT("rstd result[%ld] is: %f\n", i, resultData2[i]);
    }

    // 6. Release aclTensor and aclIntArray. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyIntArray(norm);
    aclDestroyTensor(weight);
    aclDestroyTensor(bias);
    aclDestroyTensor(out);
    aclDestroyTensor(mean);
    aclDestroyTensor(rstd);

    // 7. Release device resources.
    aclrtFree(xDeviceAddr);
    aclrtFree(weightDeviceAddr);
    aclrtFree(biasDeviceAddr);
    aclrtFree(outDeviceAddr);
    aclrtFree(meanDeviceAddr);
    aclrtFree(rstdDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}

aclnnLayerNormWithImplMode example:

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_layer_norm.h"

#define CHECK_RET(cond, return_expr) \
    do {                             \
        if (!(cond)) {               \
            return_expr;             \
        }                            \
    } while (0)

#define LOG_PRINT(message, ...)         \
    do {                                \
        printf(message, ##__VA_ARGS__); \
    } while (0)

int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
    int64_t shapeSize = 1;
    for (auto i : shape) {
        shapeSize *= i;
    }
    return shapeSize;
}

int Init(int32_t deviceId, aclrtStream* stream)
{
    // Boilerplate for resource initialization.
    auto ret = aclInit(nullptr);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
    ret = aclrtSetDevice(deviceId);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
    ret = aclrtCreateStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
int CreateAclTensorMem(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr)
{
    auto size = GetShapeSize(shape) * sizeof(T);
    // Call aclrtMalloc to allocate device-side memory.
    auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
    // Call aclrtMemcpy to copy host-side data to device-side memory.
    ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
    return 0;
}

template <typename T>
void aclCreateTensorP(const std::vector<T>& shape, void** deviceAddr, aclDataType dataType, aclTensor** tensor)
{
    // Calculate strides of a contiguous tensor.
    std::vector<int64_t> strides(shape.size(), 1);
    for (int64_t i = shape.size() - 2; i >= 0; i--) {
        strides[i] = shape[i + 1] * strides[i + 1];
    }

    // Call aclCreateTensor to create an aclTensor.
    *tensor = aclCreateTensor(
        shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
        *deviceAddr);
}

template <typename T>
void aclCreateIntArrayP(const std::vector<T>& hostData, aclIntArray** intArray)
{
    // Call the API to create an aclIntArray.
    *intArray = aclCreateIntArray(hostData.data(), hostData.size());
}

int main()
{
    // 1. (Boilerplate) Initialize the device/stream. See the acl API manual.
    // Set deviceId based on the actual device.
    int32_t deviceId = 0;
    aclrtStream stream;
    auto ret = Init(deviceId, &stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

    // 2. Construct inputs and outputs. Customize the construction based on the API.
    std::vector<int64_t> xShape = {1, 2, 32};
    std::vector<int64_t> normShape = {32};
    std::vector<int64_t> meanShape = {1, 2, 1};
    void* xDeviceAddr = nullptr;
    void* weightDeviceAddr = nullptr;
    void* biasDeviceAddr = nullptr;
    void* outDeviceAddr = nullptr;
    void* meanDeviceAddr = nullptr;
    void* rstdDeviceAddr = nullptr;
    aclTensor* x = nullptr;
    aclIntArray* norm = nullptr;
    aclTensor* weight = nullptr;
    aclTensor* bias = nullptr;
    aclTensor* out = nullptr;
    aclTensor* mean = nullptr;
    aclTensor* rstd = nullptr;
    std::vector<float> xHostData(64, 2.0);
    std::vector<int64_t> normData = {32};
    std::vector<float> weightHostData(32, 1.0);
    std::vector<float> biasHostData(32, 0.0);
    std::vector<float> outHostData(64, 0.0);
    std::vector<float> meanHostData(2, 0.0);
    std::vector<float> rstdHostData(2, 0.0);
    double eps = 1e-5;
    int32_t implMode = 0;

    // Create x aclTensor.
    ret = CreateAclTensorMem(xHostData, xShape, &xDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create weight aclTensor.
    ret = CreateAclTensorMem(weightHostData, normShape, &weightDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create bias aclTensor.
    ret = CreateAclTensorMem(biasHostData, normShape, &biasDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create out aclTensor.
    ret = CreateAclTensorMem(outHostData, xShape, &outDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create mean aclTensor.
    ret = CreateAclTensorMem(meanHostData, meanShape, &meanDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);
    // Create rstd aclTensor.
    ret = CreateAclTensorMem(rstdHostData, meanShape, &rstdDeviceAddr);
    CHECK_RET(ret == ACL_SUCCESS, return ret);

    aclCreateTensorP(xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
    aclCreateIntArrayP(normData, &norm);
    aclCreateTensorP(normShape, &weightDeviceAddr, aclDataType::ACL_FLOAT, &weight);
    aclCreateTensorP(normShape, &biasDeviceAddr, aclDataType::ACL_FLOAT, &bias);
    aclCreateTensorP(xShape, &outDeviceAddr, aclDataType::ACL_FLOAT, &out);
    aclCreateTensorP(meanShape, &meanDeviceAddr, aclDataType::ACL_FLOAT, &mean);
    aclCreateTensorP(meanShape, &rstdDeviceAddr, aclDataType::ACL_FLOAT, &rstd);

    // 3. Call the CANN operator library API. Replace with the specific API name.
    uint64_t workspaceSize = 0;
    aclOpExecutor* executor;
    // Call the first-phase API of aclnnLayerNormWithImplMode.
    ret = aclnnLayerNormWithImplModeGetWorkspaceSize(
        x, norm, weight, bias, eps, out, mean, rstd, implMode, &workspaceSize, &executor);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnLayerNormWithImplModeGetWorkspaceSize failed. ERROR: %d\n", ret);
              return ret);
    // Apply for device memory based on the workspaceSize calculated by the first-phase API.
    void* workspaceAddr = nullptr;
    if (workspaceSize > 0) {
        ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
        CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
    }
    // Call the second-phase API of aclnnLayerNormWithImplMode.
    ret = aclnnLayerNormWithImplMode(workspaceAddr, workspaceSize, executor, stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnLayerNormWithImplMode failed. ERROR: %d\n", ret); return ret);

    // 4. (Boilerplate) Synchronously wait for task execution to complete.
    ret = aclrtSynchronizeStream(stream);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

    // 5. Obtain the output value and copy the result from the device memory to the host. Modify based on the specific API.
    auto size = GetShapeSize(xShape);
    std::vector<float> resultData(size, 0);
    ret = aclrtMemcpy(
        resultData.data(), resultData.size() * sizeof(resultData[0]), outDeviceAddr, size * sizeof(resultData[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy first result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size; i++) {
        LOG_PRINT("out result[%ld] is: %f\n", i, resultData[i]);
    }

    auto size1 = GetShapeSize(meanShape);
    std::vector<float> resultData1(size1, 0);
    ret = aclrtMemcpy(
        resultData1.data(), resultData1.size() * sizeof(resultData1[0]), meanDeviceAddr, size1 * sizeof(resultData1[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy second result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size1; i++) {
        LOG_PRINT("mean result[%ld] is: %f\n", i, resultData1[i]);
    }

    auto size2 = GetShapeSize(meanShape);
    std::vector<float> resultData2(size2, 0);
    ret = aclrtMemcpy(
        resultData2.data(), resultData2.size() * sizeof(resultData2[0]), rstdDeviceAddr, size2 * sizeof(resultData2[0]),
        ACL_MEMCPY_DEVICE_TO_HOST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy last result from device to host failed. ERROR: %d\n", ret);
              return ret);
    for (int64_t i = 0; i < size2; i++) {
        LOG_PRINT("rstd result[%ld] is: %f\n", i, resultData2[i]);
    }

    // 6. Release aclTensor and aclIntArray. Modify based on the specific API definition.
    aclDestroyTensor(x);
    aclDestroyIntArray(norm);
    aclDestroyTensor(weight);
    aclDestroyTensor(bias);
    aclDestroyTensor(out);
    aclDestroyTensor(mean);
    aclDestroyTensor(rstd);

    // 7. Release device resources.
    aclrtFree(xDeviceAddr);
    aclrtFree(weightDeviceAddr);
    aclrtFree(biasDeviceAddr);
    aclrtFree(outDeviceAddr);
    aclrtFree(meanDeviceAddr);
    aclrtFree(rstdDeviceAddr);
    if (workspaceSize > 0) {
        aclrtFree(workspaceAddr);
    }
    aclrtDestroyStream(stream);
    aclrtResetDevice(deviceId);
    aclFinalize();
    return 0;
}