TransferWithCacheKeyConfig

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	x
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Constructs a TransferWithCacheKeyConfig instance.

Prototype

__init__(cache_key: Union[BlocksCacheKey, CacheKeyByIdAndIndex], src_layer_range: range = None, dst_layer_range: range = None, src_batch_index: int = 0)

Parameters

Parameter	Data Type	Description
cache_key	Union[BlocksCacheKey, CacheKeyByIdAndIndex]	Index of the remote cache.
src_layer_range	range	(Mandatory) Range of layers with data to be transmitted locally. The step must be 1.
dst_layer_range	range	(Mandatory) Range of layers with data to be transmitted remotely. The step must be 1.
src_batch_index	int	Batch index of the local cache. This parameter can be set when the source cache is in the non-PagedAttention scenario.

Example

from llm_datadist import TransferWithCacheKeyConfig
TransferWithCacheKeyConfig(BlocksCacheKey(1), range(0, 40), range(0, 40))

Returns

In normal cases, a TransferWithCacheKeyConfig instance is returned.

If a parameter is incorrect, a TypeError, a ValueError, or an LLMException may be thrown.

Restrictions

The range specified by src_layer_range must be equal to the range specified by dst_layer_range.
When cache_key is of type BlocksCacheKey, src_batch_index must be 0.

Parent topic: API Reference