Restrictions

In non-PA_ENCODER mode, the query, key, and value of Atlas A2 inference products can be two-dimensional [nTokens, hiddenSize] or four-dimensional [batch, seq_len, head_num, head_size]. In PA_ENCODER mode, the query, key, and value of Atlas A2 inference products can be two-dimensional [nTokens, hiddenSize] or three-dimensional [nTokens, head_num, head_size].
In PA_ENCODER, the number of dimensions of query, key, and value must be the same.
Atlas inference products: 0 < batch <= 2000
To use the GQA mode, the following conditions must be met: headNum > kvHeadNum; headNum%kvHeadNum == 0
In the following scenarios, head_size = head_size_v and the value ranges from 0 to 256. In other cases, head_size may not be equal to head_size_v, and the values range from 0 to 576.
- Quantization
- Use the logN scaling feature.
- inputLayout: TYPE_BNSD
- Atlas inference products: head_size must be a multiple of 16.
- Atlas training products: head_size must be a multiple of 16.
Atlas inference products: The mask size must be the actual value of max_seq_len.
Atlas inference products: q_seq_len must be equal to kv_seq_len.
nTokens in the tensor dimension:
- Atlas A2 inference products: The value is the sum of seq_len of each batch.
- Atlas inference products: In PA_ENCODER, the value is the sum of seq_len of all batches rounded up to an integer multiple of 16.
- In other cases, the value is the sum of the seq_len values of all batches rounded up to an integer multiple of 16.
How to select a mask:
- In the full phase, set maskType to MASK_TYPE_NORM or MASK_TYPE_ALIBI based on the configuration of the native model.
- In the incremental phase, if the original model is not alibi and the mask is not padded, maskType can be set to UNDEFINED, that is, the mask is not passed. In other cases, set maskType to MASK_TYPE_NORM or MASK_TYPE_ALIBI based on the configuration of the native model.
- In the long sequence scenario, you can compress the mask. For details about how to compress a mask, see compressing mask.
In PA_ENCODER mode, qScale does not take effect.
The Atlas training products support only PA_ENCODER and basic functions and do not support functions listed in Functions.
The hidden size must be a multiple of 16 in non-PA ENCODER and non-bypass scenarios.

Parent topic: SelfAttentionOperation