LLM-DataDist API List_V1

LLM-DataDist is a distributed LLM cluster and data acceleration component. It provides clustered KV data management capabilities and supports disaggregated deployment of full graphs and incremental graphs.

The following products are supported:

Atlas A2 training product/Atlas A2 inference product: For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800I A2 inference server and A200I A2 Box heterogeneous subrack are supported.
Atlas A3 training product/Atlas A3 inference product

The related APIs are stored in ${INSTALL_DIR}/include/llm_datadist/llm_datadist.h. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.

The library file corresponding to the following APIs is libllm_engine.so.

The LLM-DataDist APIs are listed as follows.

**Table 1** LLM-DataDist APIs_V1
API	Description
LlmDataDist Constructor	Constructs an LLM-DataDist instance.
~LlmDataDist()	Destructs an LLM-DataDist instance.
Initialize	Initializes the LLM-DataDist instance.
Finalize	Releases the LLM-DataDist instance.
SetRole	Sets the role of the current LLM-DataDist.
LinkLlmClusters	Establishes a link.
UnlinkLlmClusters	Disconnects a link.
PullKvCache	Pulls the KV cache in contiguous memory mode.
PullKvBlocks	Pulls the KV cache in block list mode.
CopyKvCache	Copies the KV cache in contiguous memory mode.
CopyKvBlocks	Copies the KV cache in block list mode.
PushKvCache	Pushes the cache to the remote node. This API can be called only when the role is Prompt.
PushKvBlocks	Pushes the cache to the remote node in block list mode. This API can be called only when the role is Prompt.
AllocateCache	Allocates a cache.
DeallocateCache	Deallocates a cache.

Parent topic: API Reference