enable_remote_cache_accessible

Applicability

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Specifies whether to enable the function of directly accessing the remote cache.

When this parameter is enabled, the remote cache metadata (such as index and memory address) is cached locally to accelerate the pull process. However, this also introduces relevant restrictions. For details, see Restrictions.

Prototype

1

enable_remote_cache_accessible(self, enable_remote_cache_accessible: bool)

Parameters

If this parameter is disabled, Atlas A3 training product/Atlas A3 inference product support only the RDMA transmission protocol. You are advised to enable this parameter to support more types of transmission protocols.

Example

1
2
3

from llm_datadist import LLMConfig
llm_config = LLMConfig()
llm_config.enable_remote_cache_accessible = True

Returns

In normal cases, no value is returned.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Restrictions

The index update is triggered only during the first pull. Subsequent cache operations on the remote side are not visible to the local side after that. Therefore, you need to ensure the validity of the cache. This parameter is more suitable for the PagedAttention scenario, where caches are allocated and registered only during the initialization phase of the user script and do not change frequently.