remove_cache_key

Applicability

Product	Supported (Yes/No)
Atlas 350 Accelerator Card	No
Atlas A3 training product/Atlas A3 inference product	Yes
Atlas A2 training product/Atlas A2 inference product	Yes
Atlas 200I/500 A2 inference product	No
Atlas inference product	No
Atlas training product	No

Note: For the Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Removes CacheKey. This API can be called only when LLMRole is set to PROMPT.

After a cache key is removed, the corresponding cache can no longer be pulled by pull_cache.

Prototype

remove_cache_key(cache_key: CacheKey)

Parameters

Parameter	Data Type	Description
cache_key	CacheKey	Cache key to be removed.

Example

cache_keys = [CacheKey(prompt_cluster_id=0, req_id=1)]
kv_cache_manager.remove_cache_key(cache_keys[0])

Returns

In normal cases, no value is returned.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

If the execution time exceeds the value of sync_kv_timeout, an LLMException is thrown.

Constraints

If CacheKey does not exist or has been removed, this operation is a no-operation.
This API does not support concurrent calls.

Parent topic: KvCacheManager