copy_cache

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	x
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Copies the cache.

Prototype

copy_cache(dst: Cache, src: Cache, dst_batch_index: int = 0, src_batch_index: int = 0, offset: int = 0, size: int = -1, req_id: Optional[int] = None)

Parameters

Parameter	Data Type	Description
dst	Cache	Target cache.
src	Cache	Source cache.
dst_batch_index	int	Batch index of the destination cache. The default value is 0.
src_batch_index	int	Batch index of the source cache. The default value is 0.
offset	int	Offset of each tensor. The default value is 0.
size	int	Set this parameter to an integer greater than 0, indicating the size of the copy. Or set this parameter to -1, indicating full copy. The default value is -1.
req_id	Optional[int]	Request ID associated with the current call. If this parameter is set, the request ID is recorded in the maintenance and debugging logs related to the local call. The default value is None.

Example

from llm_datadist import *
...
cache_manager.copy_cache(dst_cache, src_cache, 0, 1, 0, 128)

Returns

In normal cases, no value is returned.

If the input data type is incorrect, the TypeError or ValueError exception is reported.

If the execution time exceeds the value of sync_kv_timeout, an LLMException is thrown.

Restrictions

CacheDesc of the source cache and destination cache must be matched.

Parent topic: CacheManager (to Be Deprecated)