Common Prefixes
Function Description
A common prefix refers to the identical initial segment shared by multiple input prompts during a single inference process.
The KV cache generated from the common prefix can be copied to the KV cache of a new user request for inference.
Related APIs
API |
Description |
|---|---|
CacheManager.copy_cache |
Copies the cache in the CacheManager scenario. |
KvCacheManager.copy_cache |
Copies the cache in the KvCacheManager scenario. |
Example
1 2 3 4 | src_cache = kv_cache_manager.allocate_cache(npu_cache_desc, npu_cache_key) # Prefix cache dst_cache = kv_cache_manager.allocate_cache(npu_cache_desc, npu_cache_key) # New request cache kv_cache_manager.copy_cache(dst_cache, src_cache, dst_batch_index, src_batch_index, offset, size) |
Parent topic: Special Topics