AllocateCache

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Allocates a cache.

Prototype

Status AllocateCache(const CacheDesc &cache_desc, Cache &cache)

Parameters

Parameter	Input/Output	Description
cache_desc	Input	Cache description.
cache	Output	Allocated cache. This parameter is valid only when LLM_SUCCESS is returned.

Example

CacheDesc kv_desc{};
kv_desc.num_tensors = 80;
kv_desc.data_type = DT_FLOAT16;
kv_desc.shape = {1, 256};
Cache cache;
Status ret = llm_datadist.AllocateCache(kv_desc, cache);

Returns

LLM_SUCCESS: Success.
LLM_PARAM_INVALID: Incorrect parameter.
LLM_DEVICE_OUT_OF_MEMORY: Insufficient device memory.
Other values: Failure.

Constraints

Before calling this API, call the Initialize API to complete initialization.

This API takes effect only when placement in cache_desc is set to CachePlacement::kDevice. For details about the cache, see CacheDesc.

Parent topic: LlmDataDist APIs