RegisterKvCache

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	x
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product	x
Atlas training product	x

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Registers the local KV cache memory.

Prototype

Status RegisterKvCache(const CacheDesc &cache_desc,
                       const std::vector<uint64_t> &addrs,
                       const RegisterCfg &cfg,
                       int64_t &cache_id);

Parameters

Parameter	Input/Output	Description
cache_desc	Input	Description of the local cache.
addrs	Input	Addresses of the local cache. The number of addresses cannot exceed 240.
cfg	Input	Reserved.
cache_id	Output	ID of the registered cache. It can be used to construct a cache reference when KV transmission APIs are called.

Example

Click Gitee, download the sample package of the matching version based on the tag name, and obtain the sample from the cplusplus/level1_single_api/11_llm_data_dist directory.

Returns

LLM_SUCCESS: Success.
Other values: Failure.

Restrictions

This API must be called after the Initialize API is called to complete initialization.

If the HDK version is earlier than 25.5.0, a maximum of 20 GB host memory can be registered. If the HDK version is 25.5.0 or later, a maximum of 1 TB host memory can be registered. Larger registered memory results in higher OS memory usage. This restriction applies to the following products:

Atlas A2 training product/Atlas A2 inference product
Atlas A3 training product/Atlas A3 inference product

Parent topic: LLM-DataDist APIs