Parameters

Member	Type	Default Value	Value Range	Mandatory or Not	Description
headNum	int32_t	0	{8,16,32,64,128}	Yes	Number of query headers.
qkScale	float	1.0	(0,1]	Yes	Scaling coefficient post-multiplied by Q*K^T.
kvHeadNum	int32_t	0	[1] In the full prefill scenario: headNum==kvHeadNum	Yes	Number of KV heads
maskType	MaskType	UNDEFINED	[0,4]	No	Mask type
calcType	CalcType	CALC_TYPE_UNDEFINED	[0,4]	No	Computation type
cacheMode	CacheMode	KVCACHE	[0,3]	Yes	Input query and kcache type.
windowSize	uint32_t	0	[0, maxkvseqlen]	No	Size of the sliding window.
maskUseStatusType	MaskUseStatusType	0	[0,1]	No	Whether the mask is used. 0: MASK_USE_STATUS_TYPE_UNDEFINED 1: MASK_USE_STATUS_TYPE_BATCH_MASK
rsv[36]	uint8_t	0	[0]	No	Reserved parameter.

The user-defined types in the preceding table are described as follows.

maskType: mask type. The values are as follows:

UNDEFINED: no mask.
MASK_TYPE_SPEC: parallel decoding mask, which is used together with calcType whose value is CALC_TYPE_SPEC or CALC_TYPE_SPEC_AND_RING.

MASK_TYPE_MASK_FREE: passed mask of the static shape, which is used together with calcType whose value is CALC_TYPE_SPEC, CALC_TYPE_PREFILL, or CALC_TYPE_SPEC_AND_RING. During calculation, the actual mask is obtained from the passed mask based on the tail length. The mask is in the inverted triangle form. The first row is all -inf, and the last row is all 0. The shape is as follows:

def generate_mask_free(q_len):
        :param q_len: q_len
        :return: constructed mask, e.g. when q_len=2, returned as
        [[-inf   -inf   -inf   ...   -inf],
         [0       -inf   -inf   ...   -inf],
         [0       0       -inf   ...   -inf],
         ...
         [0       0       0       ...   -inf],
         [0       0       0       ...   0]]
        """
        mask_free = np.full((125 + 2 * q_len, 128), -10000.0)
        mask_free = np.triu(mask_free, 2 - q_len)
        return mask_free

MASK_TYPE_CAUSAL_MASK: internally generated mask.
MASK_TYPE_SWA_NORM: passed swa mask of [max(qseqlen), max(kvSeqlen)].

calcType: calculation type. The values are as follows:
- CALC_TYPE_UNDEFINED: default decoder scenario.
- CALC_TYPE_SPEC: Parallel decoding is supported when qseqlen is greater than 1. MTP is applicable to this scenario.
- CALC_TYPE_RING: ringAttention, extra output Ise.
- CALC_TYPE_SPEC_AND_RING: ringAttention with qseqlen greater than 1 can be passed.
- CALC_TYPE_PREFILL: full prefill.
cacheMode: type of the input query and kcache. The values are as follows:
This parameter is used together with the MLAPO large-fusion preprocessing operator.
- KVCACHE: The input consists of only one q and one merged kvCache, designed for the MLA scenario under the original PagedAttention, which is not yet supported.
- KROPE_CTKV: The input q is split into qNope and qRope, and the input kcache is split into ctKV and kRope, which corresponds to the previous default scenario.
- INT8_NZCACHE: high-performance cache. krope and ctkv are converted into the NZ format for output, and ctkv and qnope are converted into the int8 type through per_head static symmetric quantization, based on KROPE_CTKV.
- NZCACHE: krope and ctkv are converted into the NZ format for output, based on KROPE_CTKV.

Parent topic: MultiLatentAttentionOperation