MLA
Description
DeepSeek V2's Multi-head Latent Attention (MLA) down-projects hiddenstates to compress the hidden size and reduce the size of the KV cache. It performs PA calculation by up-projecting the KV. The throughput can be improved by exchanging the computing power for the video memory.
How to Enable
head_size_k in [num_blocks, block_size, kv_head_num, head_size_k] of keyCache is different from head_size_v in [num_blocks, block_size, kv_head_num, head_size_v] of valueCache.
Constraints
Not supported by the
Parent topic: Functions