LayerNormWithStrideOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	x
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

To enable LayerNorm to support non-contiguous tensors, the x tensor is represented by (shape, strides, offset). strides indicates the interval between two adjacent elements in a dimension, and offset indicates the offset of the first element of the tensor relative to the address.

Definition

struct LayerNormWithStrideParam {
    enum LayerNormType : int {
        LAYER_NORM_UNDEFINED = 0, 
        LAYER_NORM_NORM,
        LAYER_NORM_PRENORM,       
        LAYER_NORM_POSTNORM,     
        LAYER_NORM_MAX,
    };
    struct NormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        int32_t beginNormAxis = 0;
        int32_t beginParamsAxis = 0;
        DynamicQuantType dynamicQuantType = DYNAMIC_QUANT_UNDEFINED;
        uint8_t rsv[20] = {0};
    };
    struct PreNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    struct PostNormParam {
        QuantType quantType = QUANT_UNQUANT;
        float epsilon = 1e-5;
        uint64_t opMode = 0;
        float zoomScaleValue = 1.0f;
        uint8_t rsv[20] = {0};
    };
    LayerNormType layerType = LAYER_NORM_UNDEFINED;
    NormParam normParam;
    PreNormParam preNormParam;
    PostNormParam postNormParam;
    uint8_t rsv[8] = {0};
};

Parameters

Member	Description
layerType	Normalization type. LAYER_NORM_UNDEFINED: default, undefined. LAYER_NORM_NORM: norm.
normParam	norm parameters. For details, see Table 1.
preNormParam	PRENORM parameters. For details, see Table 2.
postNormParam	POSTNORM parameters. For details, see Table 3.
rsv[8]	Reserved.

**Table 1** normParam members
Member	Type	Default Value	Value Range	Mandatory	Description
quantType	QuantType	QUANT_UNDEFINED	QUANT_UNDEFINED	No	Quantization type. Currently, only QUANT_UNDEFINED is supported.
epsilon	float	1e-5	Greater than 0	No	Epsilon, added to the denominator during normalization to prevent division by 0.
beginNormAxis	int32_t	0	beginNormAxis must be less than the dimension size of tensor x. If the value is a negative number, the sum of the value and the tensor x dimension size must be greater than 0.	No	Normalization dimension, which specifies the start dimension of normalization and determines the dimensions of gamma and beta. Not enabled. Supports only 0.
beginParamAxis	int32_t	0	beginParamAxis must be greater than or equal to 0 and less than the dimension size of tensor x.	No	Normalization dimension, indicating the dimension from which subsequent dimensions are combined by axis. Not enabled. Supports only 0.
dynamicQuantType	DynamicQuantType	DYNAMIC_QUANT_UNDEFINED	DYNAMIC_QUANT_UNDEFINED	No	Only DYNAMIC_QUANT_UNDEFINED is supported.

**Table 2** preNormParam members
Member	Type	Default Value	Value Range	Mandatory	Description
quantType	QuantType	QUANT_UNQUANT	QUANT_UNQUANT	No	Quantization type. Currently, only QUANT_UNQUANT is supported.
epsilon	float	1e-5	Greater than 0	No	Epsilon, added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	-	No	0: high precision 1: high performance (not supported currently)
zoomScaleValue	float	1.0f	-	No	Scaling factor.
rsv[20]	uint8_t	{0}	[0]	No	Reserved.

**Table 3** postNormParam members
Member	Type	Default Value	Value Range	Mandatory	Description
quantType	QuantType	QUANT_UNQUANT	QUANT_UNQUANT QUANT_INT8	No	Quantization type. Currently, the following types are supported: QUANT_UNQUANT QUANT_INT8
epsilon	float	1e-5	Greater than 0	No	Epsilon, added to the denominator during normalization to prevent division by 0.
opMode	uint64_t	0	-	No	0: high precision 1: high performance (not supported yet)
zoomScaleValue	float	1.0f	-	No	Scaling factor.
rsv[20]	uint8_t	{0}	[0]	No	Reserved.

Input

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
gamma	[1, dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
beta	[1, dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
xStrides	[n+1]	int32	ND	Stride of x, with the value of the last dimension being 1. x_shape indicates the number of dimensions of x. The information is passed as a host tensor, with all values greater than 0, and the last value being 1.
xOffset	[1]	int32	ND	Offset of x. The offset must be greater than 0. The default value is 0. The information is passed as a host tensor.

Output

Parameter	Dimension	Data Type	Format	Description
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor with the same dimensions as x.

Constraints

Currently, the preNormParam and postNormParam members are not supported.

Parent topic: atb/infer_op_params.h