allocate_blocks_cache

Applicability

Note: For the Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server, Atlas 300I A2 inference card, and A200I A2 Box heterogeneous components are supported.

Function Description

Allocates the cache of multiple blocks in the PagedAttention scenario. After the cache is successfully allocated, you can call deallocate_blocks_cache to release the memory.

Prototype

1

allocate_blocks_cache(cache_desc: CacheDesc, blocks_cache_key: Optional[BlocksCacheKey] = None)

Parameters

Example

1
2
3
4

from llm_datadist import *
...
blocks_cache_key = BlocksCacheKey(1, 0)
blocks_cache = cache_manager.allocate_blocks_cache(cache_desc, blocks_cache_key)

Returns

In normal cases, a Cache instance is returned.

If the input data type is incorrect, a TypeError or ValueError is thrown.

If the execution time exceeds the value of sync_kv_timeout, an LLMException is thrown.

Restrictions

The memory pool must be configured before this API is called.