transfer_cache_async

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	x
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Asynchronously transmits the KV cache in a hierarchical manner.

Prototype

transfer_cache_async(self,
                     src_cache: Cache,
                     layer_synchronizer: LayerSynchronizer,
                     transfer_configs: Union[List[Union[TransferConfig, TransferWithCacheKeyConfig]], Tuple[Union[TransferConfig, TransferWithCacheKeyConfig]]],
                     src_block_indices: Optional[Union[List[int], Tuple[int]]] = None,
                     dst_block_indices: Optional[Union[List[int], Tuple[int]]] = None,
                     dst_block_memory_size: Optional[int] = None) -> CacheTask

Parameters

Parameter	Data Type	Description
src_cache	Cache	Source cache.
layer_synchronizer	LayerSynchronizer	LayerSynchronizer implementation class object.
transfer_configs	Union[List[Union[TransferConfig, TransferWithCacheKeyConfig]], Tuple[Union[TransferConfig, TransferWithCacheKeyConfig]]]	Transmission configuration list or tuple.
src_block_indices	Optional[Union[List[int], Tuple[int]]]	Block indices of the source cache, which are set when the source cache is in the PagedAttention scenario.
dst_block_indices	Optional[Union[List[int], Tuple[int]]]	Block indices of the destination cache, which are set when the destination cache is in the PagedAttention scenario.
dst_block_memory_size	Optional[int]	Size of the memory occupied by each block in the destination cache, which is set when the destination cache is in the PagedAttention scenario. If the source cache is also in the PagedAttention scenario, this parameter can be omitted. In this case, it is automatically set to the memory size occupied by each block in the source cache. This parameter is omitted if the value is set to 0.

Example

cache_task = cache_manager.transfer_cache_async(cache, LayerSynchronizerImpl(), transfer_configs)

Returns

In normal cases, a CacheTask instance is returned.

If the input data type is incorrect, the TypeError or ValueError exception is reported.

If the input data is invalid, an LLMException is reported.

Restrictions

Support is not provided when src_cache is on the host and dst_cache is on the device.
Support is not provided when src_cache is in PagedAttention mode but dst_cache is not.
Single-process multi-card scenarios are not supported.
If dst_cache is on the host, it must be allocated using the allocate_cache API.
Ensure that dst_addrs in transfer_config is valid and that data is valid when dst_block_indices is used for transmission. Otherwise, undefined errors may occur.
When enable_remote_cache_accessible is enabled, entries in transfer_configs must be of type TransferWithCacheKeyConfig. When enable_remote_cache_accessible is disabled, they must be of type TransferConfig

Parent topic: CacheManager