Parameters
The parameter members that should contain the atb::infer::SelfAttentionParam are described as follows.
Member |
Type |
Default Value |
Description |
|---|---|---|---|
quantType |
QuantType |
TYPE_QUANT_UNDEFINED |
Quant type |
outDataType |
aclDataType |
ACL_DT_UNDEFINED |
Output data type |
headNum |
int32_t |
0 |
Query head size, which must be greater than 0. |
kvHeadNum |
int32_t |
0 |
Number of KV heads. The value is determined based on the actual model. |
qScale |
float |
1 |
Query scaling coefficient |
qkScale |
float |
1 |
Tor value of the operator, which is post-multiplied by Q*K^T. |
batchRunStatusEnable |
bool |
false |
Whether to enable dynamic batch |
isTriuMask |
uint32_t |
0 |
Whether to enable the inverted triangle optimization. The optimization can be enabled only when the mask is an inverted triangle. |
calcType |
CalcType |
UNDEFINED |
Computation type |
kernelType |
KernelType |
KERNELTYPE_DEFAULT |
Kernel precision type |
clampType |
ClampType |
CLAMP_TYPE_UNDEFINED |
Clamp type |
clampMin |
float |
0 |
Minimum value of the clamp |
clampMax |
float |
0 |
Maximum value of the clamp |
maskType |
MaskType |
MASK_TYPE_UNDEFINED |
Mask type |
kvcacheCfg |
KvCacheCfg |
K_CACHE_V_CACHE |
KV cache |
scaleType |
ScaleType |
SCALE_TYPE_TOR |
Scale type |
inputLayout |
InputLayout |
TYPE_BSND |
The default data layout format is BSND. |
mlaVHeadSize |
uint32_t |
0 |
If the value is greater than 0, the function of merging KVCache with MLA is enabled, indicating that the value range of head_size of V is [0,576] when KV is merged. |
windowSize |
0 |
uint32_t |
SWA window size. The window size must be properly set. If the window size is greater than maxSeqlen of q/k, the SWA operation is degraded to the common self-attention operation. |
cacheType |
CACHE_TYPE_NORM |
CacheType |
Cache internal layout type. CACHE_TYPE_SWA indicates that the SWA KV cache optimization is enabled. Only the KV cache of the last windowSize tokens is stored. The length of the KV cache cannot exceed the value of windowSize to reduce the display memory usage. |
rsv[64] |
uint8_t |
{0} |
Reserved |
The user-defined types in the preceding table are described as follows.
- QuantType: enumeration of the quant type. The values are as follows:
- TYPE_QUANT_UNDEFINED: default value, not fused with quantization. In this case, q, k, and v are of type bf16/float16.
- TYPE_DEQUANT_FUSION: fused with dequantization. This type is reserved and cannot be used currently.
- TYPE_QUANT_QKV_OFFLINE: offline INT8 quantization. Only the
Atlas A2 inference products is supported. - TYPE_QUANT_QKV_ONLINE: online INT8 quantization. Only the
Atlas A2 inference products is supported.
- outDataType: output data type. The options are as follows:
- Only PA_ENCODER is supported, and QuantType is not TYPE_QUANT_UNDEFINED (in the format of aclDataType).
- kvHeadNum: number of KV heads. The value needs to be passed based on the actual model. The options are as follows:
- When kvHeadNum = 0, k_head_num of keyCache, v_head_num of valueCache, and num_heads of query are the same, that is, the value of num_heads.
- When kvHeadNum != 0, k_head_num of keyCache and v_head_num of valueCache are the same as kvHeadNum.
- CalcType: calculation type. The options are as follows:
- UNDEFINED: Uses both the decoder and encoder of flashAttention.
- ENCODER: Uses the encoder of FlashAttention.
- DECODER: Uses the decoder of FlashAttention.
- PA_ENCODER: Uses the encoder of PagedAttention.
- KernelType: kernel precision type. The options are as follows:
- KERNELTYPE_DEFAULT: float16 (input), float16 (batchmatmul), and float16 (output).
- KERNELTYPE_HIGH_PRECISION: float16 (input), float (batch matmul), and float16 (output).
- ClampType: clamp type. The options are as follows:
- CLAMP_TYPE_UNDEFINED: no resizing
- CLAMP_TYPE_MIN_MAX: resizes the image and specifies the maximum and minimum values.
- MaskType: mask type. The options are as follows:
- MASK_TYPE_UNDEFINED (default): mask of all 0s
- MASK_TYPE_NORM: inverted triangular mask
- MASK_TYPE_ALIBI: ALiBi mask
- MASK_TYPE_NORM_COMPRESS: inverted triangular compression mask
- MASK_TYPE_ALIBI_COMPRESS: alibi mask compression
- MASK_TYPE_ALIBI_COMPRESS_SQRT: square compression of alibi mask
- MASK_TYPE_ALIBI_COMPRESS_LEFT_ALIGN: left alignment of the alibi compression mask. It is supported only by the
Atlas A2 inference products . - MASK_TYPE_SLIDING_WINDOW_NORM: sliding window attention mask.
- MASK_TYPE_SLIDING_WINDOW_COMPRESS: sliding window attention compression mask.
- KvCacheCfg: kvcache configuration. calcType cannot be set to PA_ENCODER. The options are as follows:
- K_CACHE_V_CACHE (default): KV cache is processed.
- K_BYPASS_V_BYPASS: KV cache is directly passed.
- ScaleType: scale type. The options are as follows:
- SCALE_TYPE_TOR (default ): disables LogN scaling.
- SCALE_TYPE_LOGN: LogN scaling is used. quantType can only be 0.
- SCALE_TYPE_MAX: boundary value, which is used only to determine whether the value is out of the boundary.
- InputLayout: layout format of the input data. The options are as follows:
- The default value is TYPE_BSND. In this case, the enumerated value is 0.
- When the input format is TYPE_BNSD, the enumerated value is 1.
- CacheType: internal cache layout type. The options are as follows:
- CACHE_TYPE_NORM: normal cache.
- CACHE_TYPE_SWA: fixed-length cache.