Constraints
- blockSize can only be 128.
- batch ≤ 8192.
- For MTP, when calcType is 1 or 3 and headNum is 128, maskType cannot be 0, and qseqlen must be less than kvSeqlen.
- If cacheMode is 2 and calcType is 2 or 3, maskType is 0.
- Full prefill:
- The value of headNum must fall within [1, 128]. The value of headNum must be the same as that of kvHeadNum. maskType must be set to 0, 2, or 3, calcType must be set to 4, cacheMode must be set to 1, embeddimV must be set to 128, and the value of max(kvSeqlen) must be greater than or equal to that of max(qSeqLen).
- Only NO_MASK, normal inverted triangle mask, and swa mask are supported.
Parent topic: MultiLatentAttentionOperation