LayerNormWithStrideOperation

Applicable Products

Hardware Model	Supported or Not
Atlas 350 accelerator card	x
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

To enable LayerNorm to support non-contiguous tensors, the x tensor is represented by (shape, strides, offset). strides indicates the interval between two adjacent elements in a dimension, and offset indicates the offset of the first element of the tensor relative to the address.

Definition

struct LayerNormWithStrideParam {
    enum LayerNormType : int {
        LAYER_NORM_UNDEFINED = 0, 
        LAYER_NORM_NORM,
        LAYER_NORM_PRENORM,       
        LAYER_NORM_POSTNORM,     
        LAYER_NORM_MAX,
    };
    struct NormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        int32_t beginNormAxis = 0;
        int32_t beginParamsAxis = 0;
        DynamicQuantType dynamicQuantType = DYNAMIC_QUANT_UNDEFINED;
        uint8_t rsv[20] = {0};
    };
    struct PreNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    struct PostNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    LayerNormType layerType = LAYER_NORM_UNDEFINED;
    NormParam normParam;
    PreNormParam preNormParam;
    PostNormParam postNormParam;
    uint8_t rsv[8] = {0};
};

Parameters

Member	Description
layerType	Normalization type. LAYER_NORM_UNDEFINED: default, undefined. LAYER_NORM_NORM: norm.
normParam	norm parameters. For details, see Table 1
preNormParam	PRENORM parameters. For details, see Table 2.
postNormParam	POSTNORM parameters. For details, see Table 3.
rsv[8]	Reserved

**Table 1** normParam members
Member	Type	Default Value	Value Range	Mandatory or Not	Description
quantType	QuantType	QUANT_UNDEFINED	QUANT_UNDEFINED	No	Quantization type. Currently, only QUANT_UNDEFINED is supported.
epsilon	float	1e-5	Greater than 0	No	Epsilon, which is added to the denominator during normalization to prevent division by 0.
beginNormAxis	int32_t	0	beginNormAxis must be less than the dimension size of tensor x. If the value is a negative number, the sum of the value and the tensor x dimension size must be greater than 0.	No	Normalization dimension, which specifies the start dimension of normalization and determines the dimensions of gamma and beta. Not enabled. Only 0 is supported.
beginParamAxis	int32_t	0	beginParamAxis must be greater than or equal to 0 and less than the dimension size of tensor x.	No	Normalization dimension, indicating the dimension from which subsequent dimensions are combined by axis. Not enabled. Only 0 is supported.
dynamicQuantType	DynamicQuantType	DYNAMIC_QUANT_UNDEFINED	DYNAMIC_QUANT_UNDEFINED	No	Only DYNAMIC_QUANT_UNDEFINED is supported.

**Table 2** preNormParam members
Member	Type	Default Value	Value Range	Mandatory or Not	Description
quantType	QuantType	QUANT_UNQUANT	QUANT_UNQUANT	No	Quantization type. Currently, only QUANT_UNQUANT is supported.
epsilon	float	1e-5	Greater than 0	No	Epsilon, which is added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	-	No	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	-	No	Scaling factor
rsv[20]	uint8_t	{0}	[0]	No	Reserved

**Table 3** postNormParam members
Member	Type	Default Value	Value Range	Mandatory or Not	Description
quantType	QuantType	QUANT_UNQUANT	QUANT_UNQUANT QUANT_INT8	No	Quantization type. Currently, the following types are supported: QUANT_UNQUANT QUANT_INT8
epsilon	float	1e-5	Greater than 0	No	Epsilon, which is added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	-	No	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	-	No	Scaling factor
rsv[20]	uint8_t	{0}	[0]	No	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
gamma	[1, dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
beta	[1, dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
xStrides	[n+1]	int32	ND	Stride of x, with the value of the last dimension being 1. x_shape indicates the number of dimensions of x. The information is passed as a host tensor, with all values greater than 0, and the last value being 1.
xOffset	[1]	int32	ND	Offset of x. The offset must be greater than 0. The default value is 0. The information is passed as a host tensor.

Output

Parameter	Dimension	Data Type	Format	Description
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor with the same dimensions as x.

Constraints

Currently, the preNormParam and postNormParam members are not supported.

Parent topic: atb/infer_op_params.h