Constraints
- blockSize can only be 128.
- batch <= 8192.
- In the MTP scenario, that is, when calcType is 1 or 3 and headNum is 128, maskType cannot be 0, and qseqlen must be less than kvSeqlen.
- If cacheMode is 2 and calcType is 2 or 3, maskType is 0.
- In the full prefill scenario:
- The value of headNum must fall within [1, 128]. The value of headNum must be the same as that of kvHeadNum. maskType must be set to 0, 2, or 3, calcType must be set to 4, cacheMode must be set to 1, embeddimV must be set to 128, and the value of max(kvSeqlen) must be greater than or equal to that of max(qSeqLen).
- Only NO_MASK, normal inverted triangle mask, and swa mask are supported.
Parent topic: MultiLatentAttentionOperation