Parameters

The parameter members that should contain the atb::infer::SelfAttentionParam are described as follows.

Member	Type	Default Value	Description	Mandatory
quantType	QuantType	TYPE_QUANT_UNQUANT	Quant type	No
outDataType	aclDataType	ACL_DT_UNDEFINED	Output data type The default value indicates that the output tensor data type is automatically derived from the input tensors.	No
headNum	int32_t	0	Query head size, which must be greater than 0.	Yes
kvHeadNum	int32_t	0	Number of KV heads. The value is determined based on the actual model.	Yes
qScale	float	1	Query scaling coefficient	No
qkScale	float	1	Tor value of the operator, which is post-multiplied by Q*K^T.	Yes
batchRunStatusEnable	bool	false	Whether to enable dynamic batch	No
isTriuMask	uint32_t	0	Whether to enable the inverted triangle optimization. The optimization can be enabled only when the mask is an inverted triangle.	No
calcType	CalcType	UNDEFINED	Computation type	Yes
kernelType	KernelType	KERNELTYPE_DEFAULT	Kernel precision type KERNELTYPE_DEFAULT: Both the input/output tensor and softmax use the float16/bf16 type. KERNELTYPE_HIGH_PRECISION: The input/output tensor uses the float16/bf16 type, and softmax uses the float type.	No
clampType	ClampType	CLAMP_TYPE_UNDEFINED	Clamp type	No
clampMin	float	0	Minimum value of the clamp	No
clampMax	float	0	Maximum value of the clamp	No
maskType	MaskType	MASK_TYPE_UNDEFINED	Mask type	No
kvcacheCfg	KvCacheCfg	K_CACHE_V_CACHE	KV cache	No
scaleType	ScaleType	SCALE_TYPE_TOR	Scale type	No
inputLayout	InputLayout	TYPE_BSND	The default data layout format is BSND.	No
mlaVHeadSize	uint32_t	0	If the value is greater than 0, the function of merging KVCache with MLA is enabled, indicating that the value range of head_size of V is [0,576] when KV is merged.	No
windowSize	uint32_t	0	SWA window size. The window size must be properly set. If the window size is greater than maxSeqlen of q/k, the SWA operation is degraded to the common self-attention operation.	No
cacheType	CacheType	CACHE_TYPE_NORM	Internal cache layout type. If SWA KVCache optimization is enabled for CACHE_TYPE_SWA, only the KVCache of the last windowSize tokens is stored. The KVCache length cannot exceed windowSize to reduce the graphics memory usage.	No
rsv[64]	uint8_t	{0}	Reserved	No

The user-defined types in the preceding table are described as follows.

QuantType: enumeration of the quant type. The values are as follows:
- TYPE_QUANT_UNDEFINED: default value, not fused with quantization. In this case, q, k, and v are of type bf16/float16.
- TYPE_QUANT_UNQUANT: default value, not fused with quantization. In this case, q, k, and v are of type bf16/float16.
- TYPE_DEQUANT_FUSION: fused with dequantization. This type is reserved and cannot be used currently.
- TYPE_QUANT_QKV_OFFLINE: offline INT8 quantization. Supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- TYPE_QUANT_QKV_ONLINE: online INT8 quantization. Supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
outDataType: output data type. The value options are:
- Only PA_ENCODER is supported, and QuantType is not TYPE_QUANT_UNQUANT (in the format of aclDataType).
kvHeadNum: number of KV heads. The value needs to be passed based on the actual model. The value description is as follows:
- When kvHeadNum = 0, k_head_num of keyCache, v_head_num of valueCache, and num_heads of query are the same, that is, the value of num_heads.
- When kvHeadNum != 0, k_head_num of keyCache and v_head_num of valueCache are the same as kvHeadNum.
calcType: calculation type. The value options are:
- UNDEFINED: Uses both the decoder and encoder of flashAttention.
- ENCODER: Uses the encoder of FlashAttention.
- DECODER: Uses the decoder of FlashAttention.
- PA_ENCODER: Uses the encoder of PagedAttention.
- PREFIX_ENCODER: used when the FA supports the prefix cache.
KernelType: kernel precision type. The value options are:
- KERNELTYPE_DEFAULT: float16 (input), float16 (batchmatmul), and float16 (output).
- KERNELTYPE_HIGH_PRECISION: float16 (input), float (batch matmul), and float16 (output).
ClampType: clamp type. The value options are:
- CLAMP_TYPE_UNDEFINED: no resizing
- CLAMP_TYPE_MIN_MAX: resizes the image and specifies the maximum and minimum values.
MaskType: mask type. The value options are:
- MASK_TYPE_UNDEFINED (default): mask of all 0s
- MASK_TYPE_NORM: inverted triangular mask
- MASK_TYPE_ALIBI: ALiBi mask
- MASK_TYPE_NORM_COMPRESS: inverted triangular compressed mask. On Atlas inference products, the 2048-length window compression mask (mask dimension [1,128,2048,16]) is supported. In this case, quantization is not supported (quantType must be TYPE_QUANT_UNDEFINED), and isTriuMask is forcibly set to 1 in the source code (not required for external input at the constructor layer). scaleType must be SCALE_TYPE_TOR. For details, see Compression Mask.
- MASK_TYPE_ALIBI_COMPRESS: alibi mask compression
- MASK_TYPE_ALIBI_COMPRESS_SQRT: square compression of alibi mask
- MASK_TYPE_ALIBI_COMPRESS_LEFT_ALIGN: left alignment of the ALiBi-compressed mask. Supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- MASK_TYPE_SLIDING_WINDOW_NORM: sliding window attention mask.
- MASK_TYPE_SLIDING_WINDOW_COMPRESS: sliding window attention compressed mask.
- MASK_TYPE_CAUSAL_MASK: internally generated mask.
KvCacheCfg: KV cache configuration. calcType cannot be set to PA_ENCODER. The value options are:
- K_CACHE_V_CACHE (default): KV cache is processed.
- K_BYPASS_V_BYPASS: KV cache is directly passed.
ScaleType: scale type. The value options are:
- SCALE_TYPE_TOR (default): disables LogN scaling.
- SCALE_TYPE_LOGN: LogN scaling is used. quantType can only be 0.
- SCALE_TYPE_MAX: boundary value, which is used only to determine whether the value is out of the boundary.
InputLayout: layout format of the input data. The value options are:
- The default value is TYPE_BSND. In this case, the enumerated value is 0.
- When the input format is TYPE_BNSD, the enumerated value is 1.
CacheType: internal cache layout type. The value options are:
- CACHE_TYPE_NORM: normal cache.
- CACHE_TYPE_SWA: fixed-length cache.

Parent topic: SelfAttentionOperation