LayerNormOperation

Description

Normalizes the input data at the network layer to the [0, 1]. Currently, three normalization modes are supported: NORM, PRENORM, and POSTNORM.

Formula

Norm
Standard LayerNorm:

$\text{[math]}$

Figure 1 LayerNorm calculating formula
PostNorm
Fuses the add and layernorm operators to add x and residual and then perform layernorm normalization.

$\text{[math]}$

Figure 2 Add+LayerNorm calculating formula
PreNorm
Fuses add and layernorm operators to add x and residual and then perform layernorm normalization. The formula is the same as that of PostNorm. The difference is that this mode outputs the adding result of x and residual.

Definition

struct LayerNormParam {
    enum LayerNormType : int {
        LAYER_NORM_UNDEFINED = 0,
        LAYER_NORM_NORM,
        LAYER_NORM_PRENORM,
        LAYER_NORM_POSTNORM,
        LAYER_NORM_MAX,
    };
    struct NormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        int32_t beginNormAxis = 0;
        int32_t beginParamsAxis = 0;
        DynamicQuantType dynamicQuantType = DYNAMIC_QUANT_UNDEFINED;
        uint8_t rsv[20] = {0};
    };
    struct PreNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    struct PostNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    LayerNormType layerType = LAYER_NORM_UNDEFINED;
    NormParam normParam;
    PreNormParam preNormParam;
    PostNormParam postNormParam;
    uint8_t rsv[8] = {0};
};

Parameters

Member	Description
layerType	Normalization type. LAYER_NORM_UNDEFINED: default, undefined. LAYER_NORM_NORM: norm. LAYER_NORM_PRENORM: prenorm. LAYER_NORM_POSTNORM: postnorm.
normParam	NORM parameters. For details, see Table 1.
preNormParam	PRETNORM parameters. For details, see Table 2.
postNormParam	POSTNORM parameters. For details, see Table 3.
rsv[8]	Reserved

**Table 1** normParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNDEFINED	Quantization type. Currently, the following types are supported: QUANT_UNDEINFED; QUANT_INT8.
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by zero.
beginNormAxis	int32_t	0	Normalization dimension, indicating the dimension from which the normalization starts. The default value is 0. It also determines the input gamma and beta dimensions.
beginParamsAxis	int32_t	0	Normalization dimension, indicating the dimension from which subsequent dimensions are combined by axis. The default value is 0.
dynamicQuantType	DynamicQuantType	DYNAMIC_QUANT_UNDEFINED	Dynamic quantization type. The default value is DYNAMIC_QUANT_UNDEFINED. The current version does not support asymmetric dynamic quantization.
rsv[20]	uint8_t	{0}	Reserved

In 8.0.RC2 and later versions, normParam does not support the quantInputScale, quantInputOffset and quantInputAlpha parameters.

**Table 2** preNormParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNDEFINED	Quantization type. Currently, only QUANT_UNDEINFED is supported.
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by zero.
opMode	uint64_t	0	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	Scaling factor
rsv[20]	uint8_t	{0}	Reserved

**Table 3** postNormParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNDEFINED	Quantization type. Currently, the following types are supported: QUANT_UNDEINFED; QUANT_INT8.
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by zero.
opMode	uint64_t	0	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	Scaling factor
rsv[20]	uint8_t	{0}	Reserved

In 8.0.RC2 and later versions, postNormParam does not support the quantInputScale, quantInputOffset and quantInputAlpha parameters.

Input and Output of LAYER_NORM_NORM

Parameter	Dimension	Data Type	Format	Description
x	[-1,...,-1] The value -1 indicates that the size of the current dimension is not restricted.	float16/float/bf16	ND	Input. quantType = QUANT_INT8: The data type is float16 or bf16, and the size of the last dimension must be 32-byte aligned. quantType = QUANT_UNDEFINED: The data type is float16, float, or bf16.
gamma	[beginNormAxis:] or [1,-1] The value -1 indicates that the size of the current dimension is not restricted.	float16/float/bf16	ND	Input. quantType = QUANT_UNDEFINED: Determine other dimensions based on beginNormAxis. quantType = QUANT_INT8: The dimension range is [1,-1]. The size of the last dimension must be 32-byte aligned.
beta	[beginNormAxis:] or [1,-1] The value -1 indicates that the size of the current dimension is not restricted.	float16/float/bf16	ND	Input. quantType = QUANT_UNDEFINED: Determine other dimensions based on beginNormAxis. quantType = QUANT_INT8: The dimension range is [1,-1]. The size of the last dimension must be 32-byte aligned.
scale	[1]	float16/bf16	ND	Input. This parameter is required only when quantType is set to QUANT_INT8.
offset	[1]	int8	ND	Input. This parameter is required only when quantType is set to QUANT_INT8.
output	[-1,...,-1]	float16/float/int8/bf16	ND	Output. quantType = QUANT_INT8: The data type is int8. quantType = QUANT_UNDEFINED: The data type is float16, float, or bf16.

Input and Output of LAYER_NORM_NORM DYNAMIC_QUANT

Parameter	Dimension	Data Type	Format	Description
x	[d_0, ..., d_k, n]	float16	ND	Input. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
gamma	[1, ..., 1, n]	float16	ND	Input. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
beta	[1, ..., 1, n]	float16	ND	Input. All attributes are the same as those of gamma. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
y	[d_0, ..., d_k, n]	int8	ND	Output. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
scale	[d_0, ..., d_k]	float	ND	Output. Valid when quantType = = QUANT_INT8 and dynamicQuantType! = DYNAMIC_QUANT_UNDEFINED.
offset	[d_0, ..., d_k]	float	ND	Output. Valid when quantType == QUANT_INT8 and dynamicQuantType == DYNAMIC_QUANT_ASYMMETRIC. Not supported in the current version.

Input and Output of LAYER_NORM_PRENORM

Parameter	Dimension	Data Type	Format	Description
x	[-1,...,n] The value -1 indicates that the size of the current dimension is not restricted.	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
residual_in	[-1,...,n] The value -1 indicates that the size of the current dimension is not restricted.	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
gamma	[1,n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
beta	[1,n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
output	[-1,...,n]	float16	ND	Output.
residual_out	[-1,...,n]	float16	ND	Output. Output before Norm (equal to x + residual_in * zoom_scale).

Input and Output of LAYER_NORM_POSTNORM

Parameter	Dimension	Data Type	Format	Description
x	[-1,...,n] The value -1 indicates that the size of the current dimension is not restricted.	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
residual	[-1,...,n] The value -1 indicates that the size of the current dimension is not restricted.	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
gamma	[1,n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
beta	[1,n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
scale	[1]	float16	ND	Input. Valid when quantType = QUANT_INT8.
offset	[1]	int8	ND	Input. Valid when quantType = QUANT_INT8.
output	[-1,...,n]	float16	ND	Output.
outputQuant	[-1,...,n]	int8	ND	Output. Valid when quantType = QUANT_INT8.

Restrictions

beginNormAxis must be less than the dimension size of tensor x. If the value is a negative number, the sum of the value and the tensor x dimension size must be greater than 0.
Norm's dynamic quantization supports only DYNAMIC_QUANT_SYMMETRIC. The last dimension must be less than or equal to 12288.
Except the scale and offset tensors in the quantization scenario, the last dimensions of other tensors are the same.

Functions

Non-quantization
- Basic layernorm
- add+layernorm fusion (prenorm, postnorm)
Quantization
- layernorm quantization
- layernorm symmetric dynamic quantization
- add+layernorm quantization (postnorm)

Parent topic: atb/infer_op_params.h