aclnnAddLayerNormQuant

Applicable Products

Product	Supported
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	×
Atlas A2 training products/Atlas A2 inference products	×
Atlas 200I/500 A2 inference products	×
Atlas inference products	×
Atlas training products	×

Function

Description: The LayerNorm operator is a commonly used normalization operation in large models. The AddLayerNormQuant operator fuses the Add operator before LayerNorm and the LayerNorm normalized output to one or two downstream quantization operators, reducing data movement. The downstream quantization operator of LayerNorm can be a Quantize, AscendQuantV2, or DynamicQuant operator. The specific quantization operator type is determined by the attr parameters divMode and quantMode. When there are two downstream quantization operators, the operator types, input/output dtype combinations, and optional input combinations of the two quantization operators must be completely identical.
Formula:
$x = x1 + x2 + biasOptional$ $y = {{x-E(x)}\over\sqrt {Var(x)+epsilon}} * gamma + beta$
- When quantMode is set to "static", the outputs outScales1Out and outScales2Out are meaningless. Depending on the divMode input, the fused quantization operator may be Quantize or AscendQuantV2:
  - When divMode is set to true, the fused quantization operator is Quantize, with the following formula:
    $y1Out = round(y / scales1Optional + zeroPoints1Optional)$ $y2Out = round(y / scales2Optional + zeroPoints2Optional), \quad \text{if and only if scales2Optional is present}$
  - When divMode is set to false, the fused quantization operator is AscendQuantV2, and the formula is as follows:
    $y1Out = round(y * scales1Optional + zeroPoints1Optional)$ $y2Out = round(y * scales2Optional + zeroPoints2Optional), \quad \text{if and only if scales2Optional is present}$
- When quantMode is set to "dynamic", the inputs zeroPoints1Optional and zeroPoints2Optional have no practical significance. The fused quantization operator is DynamicQuant, and divMode is ineffective in this case:
  - If neither scales1Optional nor scales2Optional is provided, the outputs y2Out and scale2Out have no practical significance and can be ignored. The formula is as follows:
    $outScales1Out = row\_max(abs(y))/127$ $y1Out = round(y / outScales1Out)$
  - If only scales1Optional is input, the outputs y2Out and scale2Out are meaningless and can be ignored. The corresponding formula is as follows:
    $tmp1 = y * scales1Optional$ $outScales1Out = row\_max(abs(tmp1))/127$ $y1Out = round(y / outScales1Out)$
  - If both scales1Optional and scales2Optional are present, y2Out and scale2Out are valid. The corresponding formula is as follows:
    $tmp1 = y * scales1Optional, \quad tmp2 = y * scales2Optional$ $outScales1Out = row\_max(abs(tmp1))/127, \quad outScales2Out = row\_max(abs(tmp2))/127$ $y1Out = round(y / outScales1Out),\quad y2Out = round(y / outScales2Out)$
    where row_max denotes finding the maximum value of each row.

Function Prototype

Each operator uses a two-phase API. You must first call the aclnnAddLayerNormQuantGetWorkspaceSize API to obtain the input parameters and calculate the required workspace size based on the computation process, and then call the aclnnAddLayerNormQuant API to perform the computation.

aclnnStatus aclnnAddLayerNormQuantGetWorkspaceSize(
  const aclTensor *x1,
  const aclTensor *x2,
  const aclTensor *gamma,
  const aclTensor *beta,
  const aclTensor *biasOptional,
  const aclTensor *scales1Optional,
  const aclTensor *scales2Optional,
  const aclTensor *zeroPoints1Optional,
  const aclTensor *zeroPoints2Optional,
  const char      *quantMode,
  double           epsilon,
  bool             additionalOutput,
  bool             divMode,
  aclTensor       *y1Out,
  aclTensor       *y2Out,
  aclTensor       *xOut,
  aclTensor       *outScales1Out,
  aclTensor       *outScales2Out,
  uint64_t        *workspaceSize,
  aclOpExecutor  **executor)

aclnnStatus aclnnAddLayerNormQuant(
  void          *workspace,
  uint64_t       workspaceSize,
  aclOpExecutor *executor,
  aclrtStream    stream)

aclnnAddLayerNormQuantGetWorkspaceSize

Parameters

Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
x1 (aclTensor*)	Input	Represents the input for the addition operation in AddLayerNorm. The operator performs the computation x1 + x2 + biasOptional and applies Layer Normalization to the result. Corresponds to `x1` in the formula.	Supports empty tensors. When quantMode = "static", the shape supports 1 to 8 dimensions, and the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the shape supports 2 to 8 dimensions, and the data type supports FLOAT16 and BFLOAT16.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
x2 (aclTensor*)	Input	Represents the input for the addition computation in AddLayerNorm. The operator performs the calculation x1 + x2 + biasOptional and applies Layer Normalization to the result. Corresponds to `x2` in the formula.	Supports empty tensors. When quantMode = "static", the shape supports 1 to 8 dimensions, and the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the shape supports 2 to 8 dimensions, and the data type supports FLOAT16 and BFLOAT16. The shape is consistent with that of `x1`.	FLOAT32、FLOAT16、BFLOAT16	ND	1-8	√
gamma (aclTensor*)	Input	Represents the gamma parameter in layer normalization. Corresponds to `gamma` in the formula.	Supports empty tensors. When quantMode = "static", the shape supports 1 to 8 dimensions, and the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the shape supports 2 to 8 dimensions, and the data type supports FLOAT16 and BFLOAT16. The data dimensions must match the last several dimensions of `x1`/`x2`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
beta (aclTensor*)	Input	Corresponds to beta in the Layer Normalization formula, representing the beta parameter in layer normalization. Corresponds to `beta` in the formula.	Supports empty tensors. When quantMode = "static", the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the shape supports 2 to 8 dimensions, and the data type supports FLOAT16 and BFLOAT16. The shape can be consistent with `gamma`/`beta` or `x1`/`x2`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
biasOptional (aclTensor*)	Input	Optional input parameter. You can pass an aclTensor that meets the following constraints, or use nullptr as a placeholder to indicate that this optional input does not exist. Represents the input of the addition operation in AddLayerNorm. The operator performs the calculation x1 + x2 + biasOptional and applies Layer Normalization to the result. Corresponds to `biasOptional` in the formula.	Supports empty tensors. When quantMode = "static", the shape supports 1 to 8 dimensions, and the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the shape supports 2 to 8 dimensions, and the data type supports FLOAT16 and BFLOAT16. The shape can be the same as that of `gamma`/`beta` or `x1`/`x2`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
scales1Optional (aclTensor*)	Input	Optional input parameter, representing the scale/smooth input in the first fused quantization computation. Corresponds to `scales1Optional` in the formula.	Supports empty tensors. When quantMode = "static", the supported data types are FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the supported data types are FLOAT16 and BFLOAT16. The shape is consistent with `gamma`. When this optional input parameter is passed, see Constraints for value constraints.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
scales2Optional (aclTensor*)	Input	Optional input parameter, representing the scale/smooth input in the second fused quantization computation. Corresponds to `scales2Optional` in the formula.	Supports empty tensors. When quantMode = "static", the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the data type supports FLOAT16 and BFLOAT16. The shape is the same as that of `gamma`. When an optional input parameter is passed, see Constraints for value constraints.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
zeroPoints1Optional (aclTensor*)	Input	Optional input parameter, representing the zeroPoints input in the first fused quantization computation, valid only when quantMode = "static". Corresponds to `zeroPoints1Optional` in the formula.	Supports null Tensor. When quantMode = "static", the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the data type supports FLOAT16 and BFLOAT16. The shape is consistent with `gamma`. When the optional input parameter is passed, see Constraints for value constraints.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
zeroPoints2Optional (aclTensor*)	Input	Optional input parameter, representing the zeroPoints input in the second fused quantization computation, valid only when quantMode = "static". Corresponds to `zeroPoints2Optional` in the formula.	Supports empty tensor. When quantMode = "static", the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the data type supports FLOAT16 and BFLOAT16. The shape is consistent with `gamma`. When the optional input parameter is passed, see Constraints for value constraints.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
quantMode (char*)	Input	Quantization mode, which determines whether the fused operator fuses a static or dynamic quantization operator. Corresponds to `quantMode` in the formula description. The value can be "static" or "dynamic".	-	-	-	-	-
epsilon (double)	Input	Represents the epsilon in LayerNorm, a value added to the denominator to ensure numerical stability. Corresponds to `epsilon` in the formula.	-	-	-	-	-
additionalOutput (bool)	Input	Indicates whether to enable the output of x = x1 + x2 + biasOptional.	-	-	-	-	-
divMode (bool)	Input	Valid only when quantMode = "static". Indicates whether the static quantization processes scale by multiplication or division. When set to true, the operator performs division on scale during quantization computation. Corresponds to `divMode` in the formula description.	-	-	-	-	-
y1Out (aclTensor*)	Output	Represents the result of the LayerNorm output y after being quantized by the first quantization operator. Corresponds to `y1Out` in the formula.	The shape must be consistent with the input x1/x2.	INT8	ND	1-8	√
y2Out (aclTensor*)	Output	Represents the result of the LayerNorm output y after being quantized by the second quantization operator. Corresponds to `y2Out` in the formula.	The shape must be consistent with the input x1/x2.	INT8	ND	1-8	√
xOut (aclTensor*)	Output	Represents the output x of the Add result. Corresponds to `x` in the formula.	Supports empty tensors. When quantMode = "static", the data type supports FLOAT32, FLOAT16, and BFLOAT16. When quantMode = "dynamic", the data type supports FLOAT16 and BFLOAT16. The shape must be consistent with the inputs `x1`/`x2`.	FLOAT32, FLOAT16, BFLOAT16	ND	1-8	√
outScales1Out (aclTensor*)	Output	Represents the output of the first dynamic quantization outScale result, valid only when quantMode="dynamic". Corresponds to `outScales1Out` in the formula.	The shape of input `x1` with the last dimension removed.	FLOAT32	ND	0-7	√
outScales2Out (aclTensor*)	Output	Represents the output of the second path's dynamic quantization outScale result, valid only when quantMode="dynamic". Corresponds to `outScales2Out` in the formula.	The shape is the shape of the input `x1` with the last dimension removed.	FLOAT32	ND	0-7	√
workspaceSize (uint64_t*)	Output	Returns the workspace size to be applied for on the Device side.	-	-	-	-	-
executor (aclOpExecutor**)	Output	Returns the op executor, which contains the operator computation flow.	-	-	-	-	-

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

The first-phase API performs input parameter validation and returns an error in the following scenarios:

Return Code	Error Code	Description
ACLNN_ERR_PARAM_NULLPTR	161001	The input parameter is a required input, output, or required attribute, and is a null pointer.
ACLNN_ERR_PARAM_INVALID	161002	The hardware platform is not within the supported product range.
		The value of quantMode is not "static" or "dynamic".
		The input data type combination is invalid. For valid data type combinations, see Constraints below.
		The shapes of x1, gamma, and outScales1Out meet the following conditions: The dimensions of gamma are inconsistent with the last several dimensions of x1. When the quantization mode is dynamic, i.e., the input quantMode value is "dynamic", the dimension of x1 is less than 2, or the dimension of gamma is not 1. When the quantization mode is dynamic, i.e., the input quantMode value is "dynamic", the shape of outScales1Out is not the shape of input x1 with the last dimension removed.
		The shapes of all input tensors satisfy the following equality relationships: 1. The shapes of x1, x2, xOut, and y1 are not the same; when the optional input scales2Optional is present, this condition is tightened to: the shapes of x1, x2, xOut, y1, and y2 are not the same. The shapes of gamma and beta are not the same; when the optional inputs scales1Optional, scales2Optional, zeroPoints1Optional, and zeroPoints2Optional are present, their shapes differ from that of gamma. When biasOptional is present, its shape differs from both that of gamma and that of x1. When the quantization mode is dynamic, i.e., the input quantMode value is "dynamic", and scales2Optional is also present, the shape of outScales1Out differs from the shape of outScales2Out.
		The presence of optional inputs (scales1Optional, scales2Optional, zeroPoints1Optional, zeroPoints2Optional) does not satisfy the specific combination relationships.

aclnnAddLayerNormQuant

Parameters

Parameter	Input/Output	Description
workspace	Input	Address of the workspace memory allocated on the device.
workspaceSize	Input	Size of the workspace allocated on the device, obtained from the first-phase API aclnnAddLayerNormQuantGetWorkspaceSize.
executor	Input	Operator executor, which contains the operator computation flow.
stream	Input	Specifies the stream for task execution.

Return Value

aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

Functional dimensions:

The optional inputs (scales1Optional, scales2Optional, zeroPoints1Optional, zeroPoints2Optional) support the following combinations:

scales1Optional	scales2Optional	zeroPoints1Optional	zeroPoints2Optional	quantMode	Whether Valid
T	T	T	T	"static"	T
T	T	T	F	"static"	F
T	T	F	T	"static"	F
T	T	F	F	"static"	T
T	F	T	T	"static"	F
T	F	T	F	"static"	T
T	F	F	T	"static"	F
T	F	F	F	"static"	T
F	X	X	X	"static"	F
T	T	F	F	"dynamic"	T
T	F	F	F	"dynamic"	T
F	T	F	F	"dynamic"	F
F	F	F	F	"dynamic"	T
X	X	T	X	"dynamic"	F
X	X	X	T	"dynamic"	F

Where:

T indicates that the optional input is present and / indicates valid.
F indicates that the optional input is absent, and / is invalid.
X indicates that either case is valid.

Data type support description:

When quantMode is "static":

x1 Data Type	x2 Data Type	gamma Data Type	beta Data Type	bias Data Type	scale1 Data Type	scale2 Data Type	zeroPoints1 Data Type	zeroPoints2 Data Type	y1 Data Type	y2 Data Type	x Data Type	outScale1 Data Type	outScale2 Data Type
FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	INT8	INT8	FLOAT16	FLOAT32	FLOAT32
BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	INT8	INT8	BFLOAT16	FLOAT32	FLOAT32
FLOAT32	FLOAT32	FLOAT32	FLOAT32	FLOAT32	FLOAT32	FLOAT32	FLOAT32	FLOAT32	INT8	INT8	FLOAT32	FLOAT32	FLOAT32
FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT32	FLOAT32	FLOAT32	FLOAT32	INT8	INT8	FLOAT16	FLOAT32	FLOAT32
BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	FLOAT32	FLOAT32	FLOAT32	FLOAT32	INT8	INT8	BFLOAT16	FLOAT32	FLOAT32

When quantMode is "dynamic":

x1 data type	x2 data type	gamma data type	beta data type	bias data type	scale1 data type	scale2 data type	zeroPoints1 data type	zeroPoints2 data type	y1 data type	y2 data type	x data type	outScale1 data type	outScale2 data type
FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	FLOAT16	INT8	INT8	FLOAT16	FLOAT32	FLOAT32
BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	BFLOAT16	INT8	INT8	BFLOAT16	FLOAT32	FLOAT32

Deterministic computation:
aclnnAddLayerNormQuant defaults to a deterministic implementation.

Example

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_add_layer_norm_quant.h"

#define CHECK_RET(cond, return_expr)\
do {                                \
  if (!(cond)) {                    \
    return_expr;                    \
  }                                 \
} while (0)

#define LOG_PRINT(message, ...)   \
    do {                          \
  printf(message, ##__VA_ARGS__); \
} while (0)

int64_t GetShapeSize(const std::vector<int64_t> &shape) {
  int64_t shapeSize = 1;
  for (auto i : shape) {
    shapeSize *= i;
  }
  return shapeSize;
}

int Init(int32_t deviceId, aclrtStream *stream) {
  // Boilerplate. Resource initialization.
  auto ret = aclInit(nullptr);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
  ret = aclrtSetDevice(deviceId);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevicefailed. ERROR: %d\n", ret); return ret);
  ret = aclrtCreateStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
  return 0;
}

template <typename T>
int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
                    aclDataType dataType, aclTensor **tensor) {
  auto size = GetShapeSize(shape) * sizeof(T);
  // Call aclrtMalloc to apply for device memory.
  auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
  // Call aclrtMemcpy to copy data from host to device memory.
  ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);

  // Calculate the strides of a contiguous tensor.
  std::vector<int64_t> strides(shape.size(), 1);
  for (int64_t i = shape.size() - 2; i >= 0; i--) {
    strides[i] = shape[i + 1] * strides[i + 1];
  }

  // Call aclCreateTensor to create an aclTensor.
  *tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
                            shape.data(), shape.size(), *deviceAddr);
  return 0;
}

int main() {
  // 1. (Boilerplate) Initialize the device/stream. See the ACL API manual.
  // Fill in the deviceId based on your actual device.
  int32_t deviceId = 0;
  aclrtStream stream;
  auto ret = Init(deviceId, &stream);
  // Handle the check based on your own needs.
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);

  // 2. Construct the inputs and outputs. Customize the construction based on the API interface. In this example, the use case without the bias optional input and the use case with the bias input are each called once.
  float eps = 1e-6;
  bool additionalOut = true;
  bool divMode = true;
  const char* quantMode = "dynamic";

  std::vector<int64_t> xShape = {8, 64};
  std::vector<int64_t> gammaShape = {64};

  std::vector<int64_t> reduceShape = {8,};

  void *x1DeviceAddr = nullptr;
  void *x2DeviceAddr = nullptr;
  void *betaDeviceAddr = nullptr;
  void *gammaDeviceAddr = nullptr;
  void *biasDeviceAddr = nullptr;
  void *s1DeviceAddr = nullptr;
  void *s2DeviceAddr = nullptr;
  void *z1DeviceAddr = nullptr;
  void *z2DeviceAddr = nullptr;

  // Device address for the output without bias.
  void *y1DeviceAddr = nullptr;
  void *y2DeviceAddr = nullptr;
  void *xDeviceAddr = nullptr;
  void *outScales1DeviceAddr = nullptr;
  void *outScales2DeviceAddr = nullptr;

  aclTensor *x1 = nullptr;
  aclTensor *x2 = nullptr;
  aclTensor *beta = nullptr;
  aclTensor *gamma = nullptr;
  aclTensor *bias = nullptr;
  aclTensor *s1 = nullptr;
  aclTensor *s2 = nullptr;
  aclTensor *z1 = nullptr;
  aclTensor *z2 = nullptr;

  // aclTensor for the output without bias.
  aclTensor *y1 = nullptr;
  aclTensor *y2 = nullptr;
  aclTensor *x = nullptr;
  aclTensor *outScales1 = nullptr;
  aclTensor *outScales2 = nullptr;

  int64_t xShapeSize = GetShapeSize(xShape);
  int64_t gammaShapeSize = GetShapeSize(gammaShape);
  int64_t reduceShapeSize = GetShapeSize(reduceShape);

  std::vector<float> x1HostData(xShapeSize, 0x3C00);
  std::vector<float> x2HostData(xShapeSize, 0x3C00);
  std::vector<float> gammaHostData(gammaShapeSize, 0x3C00);
  std::vector<float> betaHostData(gammaShapeSize, 0x3C00);
  std::vector<float> biasHostData(gammaShapeSize, 0x3C00);

  std::vector<float> s1HostData(gammaShapeSize, 0x3C00);
  std::vector<float> s2HostData(gammaShapeSize, 0x3C00);
  std::vector<float> z1HostData(gammaShapeSize, 0x3C00);
  std::vector<float> z2HostData(gammaShapeSize, 0x3C00);

  // HostData without bias.
  std::vector<int8_t> y1HostData(xShapeSize, 0);
  std::vector<int8_t> y2HostData(xShapeSize, 0);
  std::vector<float> xHostData(xShapeSize, 0);
  std::vector<float> outScales1HostData(reduceShapeSize, 0);
  std::vector<float> outScales2HostData(reduceShapeSize, 0);

  // Create the self aclTensor.
  ret = CreateAclTensor(x1HostData, xShape, &x1DeviceAddr, aclDataType::ACL_FLOAT16, &x1);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(x2HostData, xShape, &x2DeviceAddr, aclDataType::ACL_FLOAT16, &x2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(gammaHostData, gammaShape, &gammaDeviceAddr, aclDataType::ACL_FLOAT16, &gamma);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(betaHostData,  gammaShape, & betaDeviceAddr, aclDataType::ACL_FLOAT16, &beta);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(biasHostData, gammaShape, &biasDeviceAddr, aclDataType::ACL_FLOAT16, &bias);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(s1HostData, gammaShape, &s1DeviceAddr, aclDataType::ACL_FLOAT16, &s1);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(s2HostData, gammaShape, &s2DeviceAddr, aclDataType::ACL_FLOAT16, &s2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(z1HostData, gammaShape, &z1DeviceAddr, aclDataType::ACL_FLOAT16, &z1);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(z2HostData, gammaShape, &z2DeviceAddr, aclDataType::ACL_FLOAT16, &z2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // Create the aclTensor without bias.
  ret = CreateAclTensor(y1HostData, xShape, &y1DeviceAddr, aclDataType::ACL_INT8, &y1);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(y2HostData, xShape, &y2DeviceAddr, aclDataType::ACL_INT8, &y2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT16, &x);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(outScales1HostData, reduceShape, &outScales1DeviceAddr, aclDataType::ACL_FLOAT, &outScales1);
  CHECK_RET(ret == ACL_SUCCESS, return ret);
  ret = CreateAclTensor(outScales2HostData, reduceShape, &outScales2DeviceAddr, aclDataType::ACL_FLOAT, &outScales2);
  CHECK_RET(ret == ACL_SUCCESS, return ret);

  // Example of calling the aclnnAddLayerNormQuant API, including one call with bias and one without bias.
  // 3. Call the CANN operator library API. Replace this with the specific API name.

  // 3.1 Example without the bias optional input.
  // Call the first-phase API of aclnnAddLayerNormQuant.
  uint64_t workspaceSize = 0;
  aclOpExecutor *executor;
  ret = aclnnAddLayerNormQuantGetWorkspaceSize(x1, x2, gamma, beta, bias, s1, s2, nullptr, nullptr, quantMode, eps, additionalOut, divMode, y1, y2, x, outScales1, outScales2, &workspaceSize, &executor);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddLayerNormQuantGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);

  // Apply for device memory based on the workspaceSize calculated by the first-phase API.
  void *workspaceAddr = nullptr;
  if (workspaceSize > 0) {
    ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
    CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret;);
  }
  // Call the second-phase API of aclnnAddLayerNormQuant.
  ret = aclnnAddLayerNormQuant(workspaceAddr, workspaceSize, executor, stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddLayerNormQuant failed. ERROR: %d\n", ret); return ret);

  // 4. (Boilerplate) Synchronously wait for task execution to complete.
  ret = aclrtSynchronizeStream(stream);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);

  // 5. Obtain the output value and copy the result from the device-side memory to the host-side memory. Modify based on the specific API definition.

  auto y1Size = GetShapeSize(xShape);
  std::vector<int8_t> resultDataY1(y1Size, 0);
  ret = aclrtMemcpy(resultDataY1.data(), resultDataY1.size() * sizeof(resultDataY1[0]), y1DeviceAddr, y1Size * sizeof(resultDataY1[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from Deviceto host failed. ERROR: %d\n", ret); return ret);
  LOG_PRINT("==== AddLayerNormQuant y1 output");
  for (int64_t i = 0; i < y1Size; i++) {
    LOG_PRINT("result[%ld] is: %d\n", i, resultDataY1[i]);
  }

  auto y2Size = GetShapeSize(xShape);
  std::vector<int8_t> resultDataY2(y2Size, 0);
  ret = aclrtMemcpy(resultDataY2.data(), resultDataY2.size() * sizeof(resultDataY2[0]), y2DeviceAddr, y2Size * sizeof(resultDataY2[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from Deviceto host failed. ERROR: %d\n", ret); return ret);
  LOG_PRINT("==== AddLayerNormQuant y2 output");
  for (int64_t i = 0; i < y2Size; i++) {
    LOG_PRINT("result[%ld] is: %d\n", i, resultDataY2[i]);
  }

  auto xSize = GetShapeSize(xShape);
  std::vector<float> resultDataX(xSize, 0);
  ret = aclrtMemcpy(resultDataX.data(), resultDataX.size() * sizeof(resultDataX[0]), xDeviceAddr, xSize * sizeof(resultDataX[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from Deviceto host failed. ERROR: %d\n", ret); return ret);
  LOG_PRINT("==== AddLayerNormQuant x output");
  for (int64_t i = 0; i < xSize; i++) {
    LOG_PRINT("result[%ld] is: %f\n", i, resultDataX[i]);
  }

  auto outScale1Size = GetShapeSize(reduceShape);
  std::vector<float> resultDataOutScale1(outScale1Size, 0);
  ret = aclrtMemcpy(resultDataOutScale1.data(), resultDataOutScale1.size() * sizeof(resultDataOutScale1[0]), outScales1DeviceAddr, outScale1Size * sizeof(resultDataOutScale1[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from Deviceto host failed. ERROR: %d\n", ret); return ret);
  LOG_PRINT("==== AddLayerNormQuant outScale1 output");
  for (int64_t i = 0; i < outScale1Size; i++) {
    LOG_PRINT("result[%ld] is: %f\n", i, resultDataOutScale1[i]);
  }

  auto outScale2Size = GetShapeSize(reduceShape);
  std::vector<float> resultDataOutScale2(outScale2Size, 0);
  ret = aclrtMemcpy(resultDataOutScale2.data(), resultDataOutScale2.size() * sizeof(resultDataOutScale2[0]), outScales2DeviceAddr, outScale2Size * sizeof(resultDataOutScale2[0]), ACL_MEMCPY_DEVICE_TO_HOST);
  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from Deviceto host failed. ERROR: %d\n", ret); return ret);
  LOG_PRINT("==== AddLayerNormQuant outScale2 output");
  for (int64_t i = 0; i < outScale2Size; i++) {
    LOG_PRINT("result[%ld] is: %f\n", i, resultDataOutScale2[i]);
  }


  // 6. Release aclTensor and aclScalar. Modify based on the specific API definition.
  aclDestroyTensor(x1);
  aclDestroyTensor(x2);
  aclDestroyTensor(beta);
  aclDestroyTensor(gamma);
  aclDestroyTensor(bias);
  aclDestroyTensor(s1);
  aclDestroyTensor(s2);
  aclDestroyTensor(z1);
  aclDestroyTensor(z2);

  aclDestroyTensor(y1);
  aclDestroyTensor(y2);
  aclDestroyTensor(x);
  aclDestroyTensor(outScales1);
  aclDestroyTensor(outScales2);

  // 7. Release device resources. Modify based on the specific API definition.
  aclrtFree(x1DeviceAddr);
  aclrtFree(x2DeviceAddr);
  aclrtFree(gammaDeviceAddr);
  aclrtFree(betaDeviceAddr);
  aclrtFree(biasDeviceAddr);
  aclrtFree(s1DeviceAddr);
  aclrtFree(s2DeviceAddr);
  aclrtFree(z1DeviceAddr);
  aclrtFree(z2DeviceAddr);

  aclrtFree(y1DeviceAddr);
  aclrtFree(y2DeviceAddr);
  aclrtFree(xDeviceAddr);
  aclrtFree(outScales1DeviceAddr);
  aclrtFree(outScales2DeviceAddr);

  if (workspaceSize > 0) {
    aclrtFree(workspaceAddr);
  }

  aclrtDestroyStream(stream);
  aclrtResetDevice(deviceId);
  aclFinalize();
  return 0;
}