Restrictions

Due to hardware restrictions, block_size %16 == 0. It is recommended that block_size = 128.
When parallel decoding is enabled, blockSize <= 128.
The value of each element in blockTables must be in the range of [0, num_blocks).
The value range of head_size is (0, 256]. If the format of keyCache and valueCache is NZ, the value of head_size must be a multiple of 16.
The query, keyCache, valueCache, and maskTensor input parameters support only float16 on the Atlas inference products.
Atlas inference products: 0 < batch <= 2000
In the multi-head adaptive compression, parallel decoding, quantization, and Atlas inference products scenarios, the head sizes of keyCache and valueCache are the same. The value range is (0, 256] and block_size * head_size ≤ 128 * 128. Otherwise, the head sizes of keyCache and valueCache can be different. The value range is (0, 576], when head_size of keyCache or valueCache is greater than 256, block_size is less than or equal to 128.
For the mask type, if the original model is not alibi and the mask is not padded, maskType can be set to UNDEFINED, that is, the mask is not passed. In other cases, set maskType to MASK_TYPE_NORM or MASK_TYPE_ALIBI based on the configuration of the native model.
BNSD does not support PA_ENCODER.
The IR restrictions of the Atlas training products and the Atlas inference products are the same. The Atlas training products supports only the basic function scenarios where maskType is UNDEFINED, MASK_TYPE_NORM, or MASK_TYPE_ALIBI. For other feature parameters, the default values must be used.
To use the GQA mode, the following conditions must be met: headNum > kvHeadNum and headNum%kvHeadNum == 0

Parent topic: PagedAttentionOperation