Parameters

compressType

CompressType

COMPRESS_TYPE_UNDEFINED

COMPRESS_TYPE_UNDEFINED: 0

Yes

Compression type.

• COMPRESS_TYPE_UNDEFINED: not compressed. This is the default value.
• COMPRESS_TYPE_KVHEAD: compresses the kvHead dimension of key_cache and value_cache in the ALiBi scenario.
• COMPRESS_TYPE_KVHEAD_ROPE: compresses the kvHead dimension of key_cache and value_cache in the rope scenario.

kvCacheCfg

KvCacheCfg

K_CACHE_V_CACHE

• K_CACHE_V_CACHE: 0
• K_CACHE_V_BYPASS: 1

Yes

KvCache configuration.

• K_CACHE_V_CACHE: default value. The key_cache and value_cache values are passed.
• K_CACHE_V_BYPASS: Only key_cache is passed.

rsv[16]

uint8_t

{0}

[0]

No

Reserved