copy_cache

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Copies the KV cache.

Prototype

copy_cache(dst: KvCache, src: KvCache, dst_batch_index: int = 0, src_batch_index: int = 0, offset: int = 0, size: int = -1, req_id: Optional[int] = None)

Parameters

Parameter	Data Type	Description
dst	KvCache	Target cache.
src	KvCache	Source cache.
dst_batch_index	int	Batch index of the destination cache. The default value is 0.
src_batch_index	int	Batch index of the source cache. The default value is 0.
offset	int	Offset of each tensor. The default value is 0.
size	int	If this parameter is set to an integer greater than 0, it indicates the size of the data to be copied. If this parameter is set to -1, it indicates a full copy. The default value is -1.
req_id	Optional[int]	Request ID associated with the current call. If this parameter is set, the request ID is recorded in the maintenance and debugging logs related to the local call. The default value is None.

Example

kv_cache_manager.copy_cache(dst_cache, src_cache, 0, 1, 0, 128)

Returns

In normal cases, no value is returned.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

If the execution time exceeds the value of sync_kv_timeout, an LLMException is thrown.

Constraints

CacheDesc of src/dst needs to be matched.
This API does not support concurrent calls. Concurrent calls will be queued.

Parent topic: KvCacheManager