LayerNormOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	√
Atlas A3 inference products / Atlas A3 training products	√
Atlas A2 training products / Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Normalizes the input data at the network layer to the [0, 1] range. Currently, three normalization modes are supported: NORM, PRENORM, and POSTNORM.

Formula

Norm
Standard LayerNorm:

$\text{[math]}$

Figure 1 LayerNorm calculating formula
PostNorm
Fuses the add and layernorm operators to add x and residual and then perform layernorm normalization.

$\text{[math]}$

Figure 2 Add+LayerNorm calculating formula
PreNorm
Fuses add and layernorm operators to add x and residual and then perform layernorm normalization. The formula is the same as that of PostNorm. The difference is that this mode outputs the adding result of x and residual.

Definition

      
       
         
         
           struct LayerNormParam {
    enum LayerNormType : int {
        LAYER_NORM_UNDEFINED = 0,
        LAYER_NORM_NORM,
        LAYER_NORM_PRENORM,
        LAYER_NORM_POSTNORM,
        LAYER_NORM_MAX,
    };
    struct NormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        int32_t beginNormAxis = 0;
        int32_t beginParamsAxis = 0;
        DynamicQuantType dynamicQuantType = DYNAMIC_QUANT_UNDEFINED;
        uint8_t rsv[20] = {0};
    };
    struct PreNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    struct PostNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    LayerNormType layerType = LAYER_NORM_UNDEFINED;
    NormParam normParam;
    PreNormParam preNormParam;
    PostNormParam postNormParam;
    uint8_t rsv[8] = {0};
};

          

        

      
     

Parameters

Member	Description
layerType	Normalization type. LAYER_NORM_UNDEFINED: default, undefined. LAYER_NORM_NORM: norm. LAYER_NORM_PRENORM: prenorm. LAYER_NORM_POSTNORM: postnorm. LAYER_NORM_MAX: max type.
normParam	NORM parameters. For details, see Table 1.
preNormParam	PRENORM parameters. For details, see Table 2.
postNormParam	POSTNORM parameters. For details, see Table 3.
rsv[8]	Reserved

**Table 1** normParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNQUANT	Quantization type. Currently, the following types are supported: QUANT_UNQUANT QUANT_INT8
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by 0.
beginNormAxis	int32_t	0	Normalization dimension, indicating the dimension from which the normalization starts. The default value is 0. It also determines the input gamma and beta dimensions.
beginParamsAxis	int32_t	0	Normalization dimension, indicating the dimension from which subsequent dimensions are combined by axis. The default value is 0.
dynamicQuantType	DynamicQuantType	DYNAMIC_QUANT_UNDEFINED	Dynamic quantization type. The default value is DYNAMIC_QUANT_UNDEFINED. The current version does not support asymmetric dynamic quantization.
rsv[20]	uint8_t	{0}	Reserved

In 8.0.RC2 and later versions, normParam does not support the quantInputScale, quantInputOffset and quantInputAlpha parameters.

**Table 2** preNormParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNQUANT	Quantization type. Currently, only QUANT_UNQUANT is supported.
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	Scaling factor
rsv[20]	uint8_t	{0}	Reserved

**Table 3** postNormParam members
Parameter	Type	Default Value	Description
quantType	QuantType	QUANT_UNQUANT	Quantization type. Currently, the following types are supported: QUANT_UNQUANT QUANT_INT8
epsilon	float	1e-5	Epsilon, which is added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	Scaling factor
rsv[20]	uint8_t	{0}	Reserved

In 8.0.RC2 and later versions, postNormParam does not support the quantInputScale, quantInputOffset and quantInputAlpha parameters.

Input and Output of LAYER_NORM_NORM

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/float/bf16	ND	Input. quantType = QUANT_INT8: The data type is float16 or bf16, and the size of the last dimension must be 32-byte aligned. quantType = QUANT_UNQUANT: The data type is float16, float, or bf16.
gamma	[beginNormAxis:] or [1, ..., dim_n]	float16/float/bf16	ND	Input. quantType = QUANT_UNQUANT: Determine other dimensions based on beginNormAxis. quantType = QUANT_INT8: The dimension is [dim_n]. The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n.
beta	[beginNormAxis:] or [1, ..., dim_n]	float16/float/bf16	ND	Input. quantType = QUANT_UNQUANT: Determine other dimensions based on beginNormAxis. quantType = QUANT_INT8: The dimension is [dim_n]. The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n. The dimension must be the same as that of gamma.
scale	[1]	float16/bf16	ND	Input. This parameter is required only when quantType is set to QUANT_INT8.
offset	[1]	int8	ND	Input. This parameter is required only when quantType is set to QUANT_INT8.
output	[dim_0, dim_1, ..., dim_n]	float16/float/int8/bf16	ND	Output. quantType = QUANT_INT8: The data type is int8. quantType = QUANT_UNQUANT: The data type is float16, float, or bf16.

Input and Output of LAYER_NORM_NORM DYNAMIC_QUANT

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16	ND	Input. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
gamma	[dim_n]	float16	ND	Input. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
beta	[dim_n]	float16	ND	Input. All attributes are the same as those of gamma. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
y	[dim_0, dim_1, ..., dim_n]	int8	ND	Output. The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
scale	[dim_0, dim_1, ..., dim_n-1]	float	ND	Output. Output when quantType == QUANT_INT8 and dynamicQuantType != DYNAMIC_QUANT_UNDEFINED. The x dimension must be greater than or equal to 2.
offset	[dim_0, dim_1, ..., dim_n-1]	float	ND	Output. Output when quantType == QUANT_INT8 and dynamicQuantType == DYNAMIC_QUANT_ASYMMETRIC. The dimension of x must be greater than or equal to 2. Not supported in the current version.

Input and Output of LAYER_NORM_PRENORM

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Input. The size of the last dimension must be 32-byte aligned.
residual_in	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Input. The size of the last dimension must be 32-byte aligned.
gamma	[1, ..., dim_n]	float16/bf16	ND	Input. The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n.
beta	[1, ..., dim_n]	float16/bf16	ND	Input. The size of the last dimension must be 32-byte aligned. Any number of ones can be placed before dim_n. The dimension must be the same as that of gamma.
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output.
residual_out	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output. Output before Norm (equal to x + residual_in * zoom_scale).

Input and Output of LAYER_NORM_POSTNORM

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
residual	[dim_0, dim_1, ..., dim_n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned.
gamma	[1, ..., dim_n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned. Any number of ones can be placed before dim_n.
beta	[1, ..., dim_n]	float16	ND	Input. The size of the last dimension must be 32-byte aligned. Any number of ones can be placed before dim_n. The dimension must be the same as that of gamma.
scale	[1]	float16	ND	Input. Valid when quantType = QUANT_INT8.
offset	[1]	int8	ND	Input. Valid when quantType = QUANT_INT8.
output	[dim_0, dim_1, ..., dim_n]	float16	ND	Output.
outputQuant	[dim_0, dim_1, ..., dim_n]	int8	ND	Output. Valid when quantType = QUANT_INT8.

Constraints

beginNormAxis must be less than the dimension size of tensor x. If the value is a negative number, the sum of the value and the tensor x dimension size must be greater than 0.
Norm's dynamic quantization supports only DYNAMIC_QUANT_SYMMETRIC. The last dimension must be less than or equal to 12288.
Except the scale and offset tensors in the quantization scenario, the last dimensions of other tensors are the same.
Atlas 350 accelerator cards: Only LAYER_NORM_NORM is supported, and only non-quantization use cases are supported.

Functions

Non-quantization
- Basic layernorm
- add+layernorm fusion (prenorm, postnorm)
Quantization
- layernorm quantization
- layernorm symmetric dynamic quantization
- add+layernorm quantization (postnorm)

Parent topic: atb/infer_op_params.h