Input and output

RMS_NORM_NORM

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/float/bf16	ND	The float data type is supported only when rstd is true. When rstd is true, only the float data type can be used. The size of the last dimension dim_n of all tensors must be the same. There is no restriction on the sizes of other dimensions. The size of the last dimension must be 32-byte aligned.
gamma	[1, ..., dim_n] / [dim_i, ..., dim_n] (0 <= i <= n)	float16/float/bf16	ND	The size of the last dimension must be 32-byte aligned. When normParam.rstd is true, the number of dimensions must be greater than 0 and less than the number of dimensions of x. The gamma dimension starts from the end must be the same as x. When rstd is false, the dimension is [dim_n]. The data type is the same as that of x. Any number of 1s can be placed before dim_n.
beta	[1, ..., dim_n]	float16/float/bf16	ND	This parameter is required when quantType is set to INT8. The size of the last dimension must be 32-byte aligned. The data type is the same as that of x. Any number of 1s can be placed before dim_n. The dimension must be the same as that of gamma.
scale	[1]	float16/bf16	ND	This parameter is required when quantType is set to QUANT_INT8. The data type is the same as that of x.
offset	[1]	int8	ND	This parameter is required when quantType is set to QUANT_INT8.
output	[dim_0, dim_1, ... , dim_n]	float16/float/int8/bf16	ND	If quantType is set to INT8, int8 is output.
rstd	[dim_0, ... ,1]	float16/float/bf16	ND	Output when normParam.rstd is true. If the number of dimensions of the ith dimension is greater than or equal to x minus the number of gamma dimensions, the size is 1. Otherwise, the size is the same as the size of the corresponding dimension in x. For details about an example, see Application scenario when rstd is set to true.
Except the scale, offset, and rstd parameters, the size of the last dimension n of other tensors must be the same. When quantType is set to INT8, the size of the last dimension n of other tensors must be a multiple of 32, except the scale, offset, and rstd parameters.

RMS_NORM_NORM DYNAMIC_QUANT

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16	ND	The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
gamma	[dim_n]	float16	ND	The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
beta	[dim_n]	float16	ND	The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
output	[dim_0, dim_1, ..., dim_n]	int8	ND	The size of the last dimension n must be 32-byte aligned and less than or equal to 12288.
scale	[dim_0, dim_1, ..., dim_n-1]	float	ND	Output when quantType = = QUANT_INT8 and dynamicQuantType! = DYNAMIC_QUANT_UNDEFINED.
offset	[dim_0, dim_1, ..., dim_n-1]	float	ND	Valid when quantType == QUANT_INT8 and dynamicQuantType == DYNAMIC_QUANT_ASYMMETRIC. Not supported in the current version.
When quantType is set to INT8, the size of the last dimension n of other tensors must be a multiple of 32, except the scale, offset, and rstd parameters. The setting of dynamicQuantType takes effect only when quantType is set to INT8.

RMS_NORM_PRENORM quantType = QUANT_UNQUANT

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
beta	[1, ..., dim_n]	float16/bf16	ND	Required when hasBias is set to True. The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n.
residual	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
gamma	[1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n. The dimension must be the same as that of beta.
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor. The number of dimensions and the value of each dimension are the same as those of x.
resOut	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor. The number of dimensions and the value of each dimension are the same as those of x.
The size of the last dimension n of all tensors must be the same.

RMS_NORM_PRENORM quantType = QUANT_INT8

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16	ND	The size of the last dimension must be a multiple of 32.
residual	[dim_0, dim_1, ..., dim_n]	float16	ND	The size of the last dimension must be a multiple of 32.
gamma	[1, ..., dim_n]	float16	ND	The size of the last dimension must be a multiple of 32. Any number of 1s can be placed before dim_n.
beta	[1, ..., dim_n]	float16	ND	The size of the last dimension must be a multiple of 32. Any number of 1s can be placed before dim_n. The dimension must be the same as that of gamma.
scale	[1]	float16	ND	Quantization scale
offset	[1]	int8	ND	Quantization offset
outputQuant	[dim_0, dim_1, ..., dim_n]	int8	ND	Output tensor. The number of dimensions and the value of each dimension are the same as those of x.
output	[dim_0, dim_1, ..., dim_n]	float16	ND	Output tensor. The number of dimensions and the value of each dimension are the same as those of x.
Except the scale and offset parameters, the size of the last dimension n of other tensors must be the same.

RMS_NORM_POSTNORM quantType = QUANT_UNQUANT

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Currently, quantType = QUANT_UNQUANT is supported. The size of the last dimension must be 32-byte aligned.
beta	[1, ..., dim_n]	float16/bf16	ND	Required when hasBias is set to True. The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n.
residual	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned.
gamma	[1, ..., dim_n]	float16/bf16	ND	The size of the last dimension must be 32-byte aligned. Any number of 1s can be placed before dim_n. The dimension must be the same as that of beta.
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor. The number of dimensions and the value of each dimension are the same as those of x.
The size of the last dimension n of all tensors must be the same.

RMS_NORM_POSTNORM quantType = QUANT_INT8

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Input tensor 0. The size of the last dimension must be a multiple of 32. The size of n of the last dimension of all tensors must be the same.
residual	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Input tensor 1. The size of the last dimension must be a multiple of 32.
gamma	[dim_n]/[1, dim_n]	float16/bf16	ND	Input tensor 2. The size of the last dimension must be a multiple of 32.
scale	[1]	float16/bf16	ND	Input tensor 3. Quantization scale.
offset	[1]	int8	ND	Input tensor 4. Quantization offset.
outputQuant	[dim_0, dim_1, ..., dim_n]	int8	ND	Output tensor 0. The number of dimensions and the value of each dimension are the same as those of x. The size of the last dimension must be a multiple of 32.
output	[dim_0, dim_1, ..., dim_n]	float16/bf16	ND	Output tensor 1. The number of dimensions and the value of each dimension are the same as those of x. The size of the last dimension must be a multiple of 32.
Except the scale and offset parameters, the size of the last dimension n of other tensors must be the same.

Parent topic: RmsNormOperation