Enabling the Single-input Single-output Function
kvCacheCfg=K_CACHE_V_BYPASS, compressType=COMPRESS_TYPE_UNDEFINED
Input
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
key |
[num_tokens, k_num_head, k_head_size] |
float16/bf16/int8 |
ND |
Keys of multiple tokens in the current step. |
keyCache |
[num_blocks, block_size, k_num_head, k_head_size] |
float16/bf16/int8 |
ND |
Key cache of the current layer. |
slotMapping |
[num_tokens] |
int32 |
ND |
Storage offset of each token key in the cache, that is, (block_id × block_size + offset_in_block). The value range is (INT32_MIN, num_blocks × block_size) and no duplicate value exists. If the value is a negative number, no movement operation is performed. |
Output
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
keyCacheOut |
[num_blocks, block_size, k_num_head, k_head_size] |
float16/bf16/int8 |
ND |
All key caches. The address is the same as that of the input keyCache. |