Parameters

compressType

CompressType

COMPRESS_TYPE_UNDEFINED

0, 1, 2

No

Compression type.

• COMPRESS_TYPE_UNDEFINED: not compressed. This is the default value.
• COMPRESS_TYPE_KVHEAD: compresses the kvHead dimension of key_cache and value_cache.
• COMPRESS_TYPE_KVHEAD_ROPE: compresses the kvHead dimension of key_cache and value_cache in the rope scenario.

kvCacheCfg

KvCacheCfg

K_CACHE_V_CACHE

0, 1, 2

No

KvCache configuration.

• K_CACHE_V_CACHE: default value. The key_cache and value_cache values are passed.
• K_CACHE_V_BYPASS: Only key_cache is passed.
• K_CACHE_V_CACHE_NZ: key_cache and value_cache are passed, and the NZ format is used.

rsv[16]

uint8_t

{0}

[0]

No

Reserved.

On Atlas 350 accelerator cards, compressType supports only COMPRESS_TYPE_UNDEFINED, and kvCacheCfg supports K_CACHE_V_CACHE and K_CACHE_V_BYPASS.