SwigluQuantOperation

Applicable Products

Hardware Model	Supported or Not
Atlas 350 accelerator card	x
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Activates the swiglu function to perform per-token quantization on the output.

Definition

struct SwigluQuantParam{
    enum QuantType:int{        
        QUANT_TYPE_PER_TOKEN =0,
    };    
    QuantType quantType = QUANT_TYPE_PER_TOKEN;
    uint8_t rsv[8] = {0};
};

Parameters

Member	Type	Default Value	Value Range	Mandatory or Not	Description
quantType	QuantType	QUANT_TYPE_PER_TOKEN	[0]	Yes	Per-token quantization.
rsv[8]	uint8_t	{0}	[0]	No	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
inTensor	[nTokens, 2 * hiddenSize]	float16/bf16	ND	Input tensor

Output

Parameter	Dimension	Data Type	Format	Description
outTensor1	[nTokens, hiddenSize]	int8	ND	Output tensor, quantization output.
outTensor2	[nTokens]	float	ND	Output tensor, quantized scale.

Constraints

25 × hiddenSize (number of digits after 32-byte alignment) < 192 KB

Parent topic: atb/infer_op_params.h