Parameters

The parameter members that should contain the namespace atb::infer::PagedAttentionParam are described as follows.

Member	Type	Default Value	Description
headNum	int32_t	0	Size of the query head. This default value is unavailable. You must set this parameter.
qkScale	float	1.0	Operator tor value.
kvHeadNum	int32_t	0	Number of KV heads.
maskType	MaskType	UNDEFINED	Mask type.
batchRunStatusEnable	bool	false	Whether to enable calculable batch.
quantType	QuantType	TYPE_QUANT_UNQUANT	Quantization type.
outDataType	aclDataType	ACL_DT_UNDEFINED	Output data type. The default value indicates that the output tensor data type is automatically derived from the input tensors.
hasQuantOffset	bool	false	Whether to use the offset after the quantization function is enabled.
compressType	CompressType	COMPRESS_TYPE_UNDEFINED	Compression type.
calcType	CalcType	CALC_TYPE_UNDEFINED	Computation type.
scaleType	ScaleType	SCALE_TYPE_TOR	Scale type: When the value is SCALE_TYPE_LOGN, quanttype cannot be 2 or 3.
inputLayout	InputLayout	TYPE_BSND	The default data layout format is BSND.
mlaVHeadSize	uint32_t	0	If the value is greater than 0, the function of merging KV cache with MLA is enabled, indicating that the value range of head_size of V is [0,576] when KV is merged.
qScale	float	0	Query scaling coefficient.
rsv[64]	uint8_t	{0}	Reserved

The user-defined types in the preceding table are described as follows.

maskType: enumeration of the mask type. The values are as follows. The dimension varies according to the value of MaskType or the running hardware. For details, see Mask Types.
- UNDEFINED (enumerated value: 0): all-zero mask, default value.
- MASK_TYPE_NORM (enumerated value: 1): inverted triangle mask.
- MASK_TYPE_ALIBI (enumerated value: 2): alibi mask.
- MASK_TYPE_SPEC (enumerated value: 3): parallel decoding mask.
- MASK_TYPE_MASK_FREE (enumerated value: 4): mask_free. Only float16 is supported.
- MASK_TYPE_NORM_COMPRESS (enumerated value: 5): compressed mask. Supported on Atlas inference products. A fixed 2048 x 2048 inverted triangular compressed mask is used (the lower triangle and diagonal are 0/visible, and the upper triangle excluding the diagonal is a large negative value/invisible. -60000 or -10000 is recommended). The mask dimension is [2048, 2048] (ND). For details, see the mask type. When using this mask, quantType must be TYPE_QUANT_UNQUANT, and calcType must be CALC_TYPE_SPEC. For details, see Constraints. scaleType must be SCALE_TYPE_TOR.
quantType: enumeration of the quantization type. The values are as follows:
- TYPE_QUANT_UNDEFINED (enumerated value: 0): not fused with quantization. This is the default value.
- TYPE_QUANT_UNQUANT (enumerated value: 0): not fused with quantization. This is the default value.
- TYPE_DEQUANT_FUSION (enumerated value: 1): fused with dequantization, supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- TYPE_QUANT_QKV_OFFLINE (enumerated value: 2): offline INT8 quantization, supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- TYPE_QUANT_QKV_ONLINE (enumerated value: 3): online INT8 quantization, supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- Restrictions on the quantization types:
  - TYPE_QUANT_UNQUANT: q, keyCache, and valueCache are of type bf16/float16.
  - TYPE_DEQUANT_FUSION: q is of type bf16/float16, while keyCache and valueCache are of type int8.
  - TYPE_QUANT_QKV_OFFLINE or TYPE_QUANT_QKV_ONLINE: q, keyCache, and valueCache are of type int8.
  - The headsize of keyCache and valueCache has the same length. The value range is (0, 256], and block_size × head_size ≤ 128 × 128.
  - TYPE_QUANT_QKV_OFFLINE/TYPE_QUANT_QKV_ONLINE: outDataType must be ACL_FLOAT16 or ACL_BF16. inputLayout supports only TYPE_BSND.
compressType: enumeration of the compression type. The values are as follows:
- COMPRESS_TYPE_UNDEFINED (enumerated value: 0): not compressed. This is the default value.
- COMPRESS_TYPE_KVHEAD (enumerated value: 1): compresses the kvHead dimension of key_cache and value_cache. This value is supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- COMPRESS_TYPE_KVHEAD_ROPE (enumerated value: 2): compresses the kvHead dimension of key_cache and value_cache in RoPE. This value is supported only by Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
- COMPRESS_TYPE_MAX (enumerated value: 3): boundary of the compression type, which is used only to determine whether the value is out of the boundary. This value cannot be used in all cases.
- Restrictions on the compression types:
  - COMPRESS_TYPE_KVHEAD: quanttype cannot be 2 or 3.
  - COMPRESS_TYPE_KVHEAD_ROPE: maskType must be set to 0. In this case, quanttype cannot be 2 or 3.
calcType: enumeration of the calculation type. The values are as follows:
- CALC_TYPE_UNDEFINED (enumerated value: 0): Parallel decoding is disabled. This is the default value.
- CALC_TYPE_SPEC (enumerated value: 1): parallel decoding. In this case, only quantType = 0 is supported.
scaleType: whether to enable LogN. The values are as follows:
- SCALE_TYPE_TOR (enumerated value: 0): disables LogN scaling. This is the default value.
- SCALE_TYPE_LOGN (enumerated value: 1): LogN scaling is used for the attention.
- SCALE_TYPE_MAX (enumerated value: 2): boundary, which is used only to determine whether the value is out of the boundary.
- Restrictions on the scale types:
  SCALE_TYPE_LOGN: quanttype cannot be 2 or 3.
inputLayout: enumeration of the data layout format. The values are as follows:
- TYPE_BSND (enumerated value: 0): default value, indicating that the data layout is BSND.
- TYPE_BNSD (enumerated value: 1): The data layout is BNSD.

Parent topic: PagedAttentionOperation