create_cpu_cache

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Creates a CPU cache.

Prototype

create_cpu_cache(cache_desc: CacheDesc, addrs: Union[List[int], List[List[int]]])

Parameters

Parameter	Data Type	Description
cache_desc	CacheDesc	Cache description.
addrs	Union[List[int], List[List[int]]]	Address of the CPU cache.

Example

from llm_datadist import KvCache
cpu_addrs = []
# Configure the addresses of multiple devices in single-process multi-device mode.
# cpu_addrs = [[]]
cpu_cache = KvCache.create_cpu_cache(cpu_cache_desc, cpu_addrs) # cpu_addrs is obtained from the created CPU tensors.

Returns

In normal cases, a CPU cache of the KvCache type is returned.

If the input data type is incorrect, a TypeError or ValueError is thrown.

If the input parameter is None, an AttributeError is thrown.

Constraints

None

Parent topic: KvCache