init

Applicability

Note: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.

Function Description

Initializes the LLM-DataDist instance. You need to set the CacheManager mode in options of init, that is, set enable_cache_manager to True or specify local_comm_res.

In this mode, Decode and Prompt can bidirectionally pull the KV cache.

Prototype

1

init(options: Dict[str, str])

Parameters

Example

1
2
3
4
5
6
7

from llm_datadist import LLMDataDist, LLMRole, LLMConfig
llm_datadist = LLMDataDist(LLMRole.PROMPT, 0)
llm_config = LLMConfig()
llm_config.enable_cache_manager = True
llm_config.device_id = 0
engine_options = llm_config.generate_options()
llm_datadist.init(engine_options)

Returns

In normal cases, no value is returned.

If an exception occurs, an LLMException is thrown.

If a parameter is incorrect, a TypeError or ValueError may be thrown.

Restrictions

After the initialization is successful, you need to call finalize before the system exits.