Parameters
The parameter members that should contain the namespace atb::infer::PagedAttentionParam are described as follows.
Member |
Type |
Default Value |
Description |
|---|---|---|---|
headNum |
int32_t |
0 |
Size of the query head. |
qkScale |
float |
1.0 |
Tor value of the operator, which is post-multiplied by Q*K^T. |
kvHeadNum |
int32_t |
0 |
Number of KV heads |
maskType |
MaskType |
UNDEFINED |
Mask type |
batchRunStatusEnable |
bool |
false |
Whether to enable calculable batch |
quantType |
QuantType |
TYPE_QUANT_UNDEFINED |
Quantization type |
outDataType |
aclDataType |
ACL_DT_UNDEFINED |
Output data type |
hasQuantOffset |
bool |
false |
Whether to use the offset after the quantization function is enabled |
compressType |
CompressType |
COMPRESS_TYPE_UNDEFINED |
Compression type |
calcType |
CalcType |
CALC_TYPE_UNDEFINED |
Computation type |
scaleType |
ScaleType |
SCALE_TYPE_TOR |
Scale type: When the value is SCALE_TYPE_LOGN, quanttype cannot be 2 or 3. |
inputLayout |
InputLayout |
TYPE_BSND |
The default data layout format is BSND. |
mlaVHeadSize |
uint32_t |
0 |
If the value is greater than 0, the function of merging KV cache with MLA is enabled, indicating that the value range of head_size of V is [0,576] when KV is merged. |
rsv[68] |
uint8_t |
{0} |
Reserved |
The user-defined types in the preceding table are described as follows.
- MaskType: enumeration of the mask type. The options are as follows. The dimension varies according to the value MaskType or the running hardware. For details, see mask types.
- UNDEFINED (enumerated value: 0): mask of all 0s, default value.
- MASK_TYPE_NORM (enumerated value: 1): inverted triangle mask.
- MASK_TYPE_ALIBI (enumerated value: 2): alibi mask.
- MASK_TYPE_SPEC (enumerated value: 3): parallel decoding mask.
- QuantType: enumeration of the quantization type. The options are as follows:
- TYPE_QUANT_UNDEFINED (enumerated value: 0): not fused with quantization. This is the default value.
- TYPE_DEQUANT_FUSION (enumerated value: 1): fused with dequantization. Only the
Atlas A2 inference products is supported. - TYPE_QUANT_QKV_OFFLINE (enumerated value: 2): offline INT8 quantization. Only the
Atlas A2 inference products is supported. - TYPE_QUANT_QKV_ONLINE (enumerated value: 3): online INT8 quantization. Only the
Atlas A2 inference products is supported. - Restrictions on the quantization types:
- TYPE_QUANT_UNDEFINED: q, keyCache, and valueCache are of type bf16/float16.
- TYPE_DEQUANT_FUSION: q is of type bf16/float16, while keyCache and valueCache are of type int8.
- TYPE_QUANT_QKV_OFFLINE/TYPE_QUANT_QKV_ONLINE: q, keyCache, and valueCache are of type int8.
- The headsize of keyCache and valueCache has the same length. The value range is (0, 256], and block_size * head_size ≤ 128 * 128.
- TYPE_QUANT_QKV_OFFLINE/TYPE_QUANT_QKV_ONLINE: outDataType must be ACL_FLOAT16 or ACL_BF16. inputLayout supports only TYPE_BSND.
- CompressType: enumeration of the compression type. The options are as follows:
- COMPRESS_TYPE_UNDEFINED (enumerated value: 0): not compressed. This is the default value.
- COMPRESS_TYPE_KVHEAD (enumerated value: 1): compresses the kvHead dimension of key_cache and value_cache. Only the
Atlas A2 inference products is supported. - COMPRESS_TYPE_KVHEAD_ROPE (enumerated value: 2): compresses the kvHead dimension of key_cache and value_cache in the rope scenario. Only the
Atlas A2 inference products is supported. - COMPRESS_TYPE_MAX (enumerated value: 3): boundary of the compression type, which is used only to determine whether the value is out of the boundary. This value cannot be used in all cases.
- Restrictions on the compression types:
- COMPRESS_TYPE_KVHEAD: quanttype cannot be 2 or 3.
- COMPRESS_TYPE_KVHEAD_ROPE: maskType must be set to 0. In this case, quanttype cannot be 2 or 3.
- CalcType: enumeration of the calculation type. The options are as follows:
- CALC_TYPE_UNDEFINED (enumerated value: 0): Parallel decoding is disabled. This is the default value.
- CALC_TYPE_SPEC (enumerated value: 1): parallel decoding. In this case, only quantType = 0 is supported.
- ScaleType: whether to enable the LogN feature. The options are as follows:
- SCALE_TYPE_TOR (enumerated value: 0): disables LogN scaling. This is the default value.
- SCALE_TYPE_LOGN (enumerated value: 1): LogN scaling is used for the attention.
- SCALE_TYPE_MAX (enumerated value: 2): boundary, which is used only to determine whether the value is out of the boundary.
- Restrictions on the scale types:
- InputLayout: enumeration of the data layout format. The options are as follows:
- TYPE_BSND (enumerated value: 0): default value, indicating that the data layout is BSND.
- TYPE_BNSD (enumerated value: 1): The data layout is BNSD.