Disabling the Multi-Head Compression Function

compressType=COMPRESS_TYPE_UNDEFINED, kvCacheCfg=K_CACHE_V_CACHE

Input

Parameter	Dimension	Data Type	Format	Description
key	[num_tokens, num_head, k_head_size]	float16/bf16/int8	ND	Key of multiple tokens in the current step. If the multi-head compression function is disabled, the last dimension of the key can be different from that of value.
value	[num_tokens, num_head, v_head_size]	float16/bf16/int8	ND	Value of multiple tokens in the current step. If the multi-head compression function is disabled, the last dimension of the key can be different from that of value.
keyCache	[num_blocks, block_size, num_head, k_head_size]	float16/bf16/int8	ND/NZ	Key cache of the current layer. When multi-head compression is disabled, the last dimension of keyCache can be different from that of valueCache.
valueCache	[num_blocks, block_size, num_head, v_head_size]	float16/bf16/int8	ND/NZ	Value cache of the current layer. When multi-head compression is disabled, the last dimension of keyCache can be different from that of valueCache.
slotMapping	[num_tokens]	int32	ND	Storage offset of each token key or value in the cache, that is, (block_id × block_size + offset_in_block). The value range is (INT32_MIN, num_blocks × block_size) and no duplicate value exists. If the value is a negative number, no movement operation is performed.

If the NZ format needs to be used on the Atlas A2 training products/Atlas A2 inference products, see Enabling the Function of Converting the Format of the Input to NZ.
Atlas A2 training products/Atlas A2 inference products and Atlas 350 accelerator card: keyCache and valueCache support only the ND format.
Atlas inference products: keyCache and valueCache support only the NZ format. For other restrictions, see Constraints.

Output

Parameter	Dimension	Data Type	Format	Description
keyCacheOut	[num_blocks, block_size, num_head, k_head_size]	float16/bf16/int8	ND/NZ	All key caches. The address is the same as that of the input keyCache.
valueCacheOut	[num_blocks, block_size, num_head, v_head_size]	float16/bf16/int8	ND/NZ	All value caches. The address is the same as that of valueCache.

Parent topic: Input and Output