PullKvBlocks

Applicable Products

Product	Supported (√/x)
Ascend 950PR/Ascend 950DT	√
Atlas A3 training products/Atlas A3 inference products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	x
Atlas inference products	x
Atlas training products	x

Note: For Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Pulls the cache from the remote node to the local cache through a block list.

Prototype

Status PullKvBlocks(const CacheIndex &src_cache_index,
                    const Cache &dst_cache,
                    const std::vector<uint64_t> &src_blocks,
                    const std::vector<uint64_t> &dst_blocks,
                    const KvCacheExtParam &ext_param = {})

Parameters

Parameter	Input/Output	Description
src_cache_index	Input	Index of the remote source cache.
dst_cache	Input	Local destination cache. Only cache_id returned by calling RegisterKvCache needs to be specified.
src_blocks	Input	Block index list of the remote source cache.
dst_blocks	Input	Block index list of the local destination cache.
ext_param	Input	The difference between second and first in src_layer_range of ext_param must equal that in dst_layer_range. The default values of first and second in both src_layer_range and dst_layer_range are -1, indicating all layers. The valid range for each index is [0, maximum available layer index], and first must be less than or equal to second. The maximum available layer index is calculated as follows: (CacheDesc::num_tensors / KvCacheExtParam::tensor_num_per_layer) - 1 tensor_num_per_layer can range from 1 to the total number of tensors in the cache, with a default value of 2. When src_layer_range or dst_layer_range uses non-default values, tensor_num_per_layer can either remain at its default value or be set to another value, which must be divisible by the total number of tensors in the cache.

Returns

LLM_SUCCESS: Success.
LLM_PARAM_INVALID: Incorrect parameter.
LLM_NOT_YET_LINK: No link is established with the remote cluster.
LLM_TIMEOUT: Pull timeout.
LLM_KV_CACHE_NOT_EXIST: The remote KV cache does not exist.
Other values: Failure.

Constraints

Before calling this API, call the LinkLlmClusters API to complete initialization.

Parent topic: LLM-DataDist APIs