Constraints

Due to hardware restrictions, block_size %16 == 0. It is recommended that block_size = 128.
When parallel decoding is enabled, block_size ≤ 128.
The value of each element in blockTables must be in the range of [0, num_blocks).
The value range of head_size is (0, 256]. If the format of keyCache and valueCache is NZ, the value of head_size must be a multiple of 16.
On Atlas inference products:
- The query, keyCache, valueCache, and maskTensor input parameters support only float16.
- 0 < batch ≤ 2048.
- When maskType is set to 4, the following constraints apply:
  - block_size = 128, qseqlen ≤ kvsqlen, (kvsqlen - qseqlen) % 128 = 0 (block_size = 128).
  - calcType = CALC_TYPE_SPEC.
- When maskType is set to 5 (MASK_TYPE_NORM_COMPRESS), the following constraints apply:
  - The mask data type can only be float16.
  - Quantization is not supported. quantType must be TYPE_QUANT_UNQUANT.
  - calcType = CALC_TYPE_SPEC (parallel decoding/splitfuse).
  - scaleType = SCALE_TYPE_TOR (SCALE_TYPE_LOGN is not supported. The value is hardcoded to TOR in the op-plugin.)
  - Mask dimensions: ND = [2048, 2048].
  - The mask is a 2048 x 2048 shared inverted triangular compressed mask. All batches/heads share the same mask, and only one copy needs to be prepared. The mask is a standard causal mask: The upper triangle does not contain the diagonal, filled with a large negative value (invisible; -60000 or -10000 is recommended, and -10000 is used in the op-plugin test). The lower triangle and diagonal are 0 (visible).
  - kvSeqLen (that is, contextLens) must meet the condition: kvSeqLen ≥ qSeqLen. kvSeqLen indicates the sum of the historical KV length and the current query length (not the historical length only). The operator automatically calculates the prefix length (prefix = max(0, kvSeqLen – qSeqLen)) based on kvSeqLen and qSeqLen for mask alignment. No additional parameters need to be passed.
  - The sequence length can exceed 2048. The operator automatically adapts to the length, requiring no additional operations from the user.
In the multi-head adaptive compression, parallel decoding, Atlas inference products, and quantization scenarios, the head sizes of keyCache and valueCache are the same. The value range is (0, 256] and block_size × head_size ≤ 128 × 128. Otherwise, the head sizes of keyCache and valueCache can be different. The value range is (0, 576], when head_size of keyCache or valueCache is greater than 256, block_size is less than or equal to 128.
For the mask type, if the original model is not alibi and the mask is not padded, maskType can be set to UNDEFINED, that is, the mask is not passed. In other cases, set maskType to MASK_TYPE_NORM or MASK_TYPE_ALIBI based on the configuration of the native model.
The input and output restrictions of Atlas training products and Atlas inference products are the same. The Atlas training products support only the basic functions when maskType is UNDEFINED, MASK_TYPE_NORM, or MASK_TYPE_ALIBI. For other feature parameters, use the default values.
To use the GQA mode, the following conditions must be met: headNum > kvHeadNum and headNum%kvHeadNum == 0
When block_size = head_size = 256, the input format must meet the BSND, non-quantization, and decoder requirements on Atlas A2 training products/Atlas A2 inference products.
max_num_blocks_per_seq >= (max(contextLens) + block_size - 1) // block_size.
Atlas 350 accelerator cards support only basic use cases.

Parent topic: PagedAttentionOperation