SwigluQuantOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	√
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

This SwiGLU activation function performs per-token quantization on the output.

Definition

struct SwigluQuantParam{
    enum QuantType:int{        
        QUANT_TYPE_PER_TOKEN =0,
    };    
    QuantType quantType = QUANT_TYPE_PER_TOKEN;
    uint8_t rsv[8] = {0};
};

Parameters

Member	Type	Default Value	Value Range	Mandatory	Description
quantType	QuantType	QUANT_TYPE_PER_TOKEN	[0]	Yes	Per-token quantization.
rsv[8]	uint8_t	{0}	[0]	No	Reserved.

Input

Parameter	Dimension	Data Type	Format	Description
inTensor	[nTokens, 2 * hiddenSize]	float16/bf16	ND	Input tensor.

Output

Parameter	Dimension	Data Type	Format	Description
outTensor1	[nTokens, hiddenSize]	int8	ND	Output tensor, quantization output.
outTensor2	[nTokens]	float	ND	Output tensor, quantized scale.

Constraints

Atlas A3 inference products/Atlas A3 training products and Atlas A2 training products/Atlas A2 inference products: hiddenSize ≤ 7840
Atlas 350 accelerator cards: hiddenSize ≤ 5072

Parent topic: atb/infer_op_params.h