LLM-DataDist Application Scenarios

LLM-DataDist enables a disaggregated Prefill-Decode (PD) architecture for LLM inference.

In LLM inference, the user prompt is fed into the model during the Prefill phase, where the model computes intermediate results, writes them to the KV cache, and generates the first token. In the Decode phase, this first token is fed back into the model, which reads the previously generated KV cache from the memory to continue computation. For more details on KV cache-based LLM inference, see LLM Inference Process.

To improve performance and resource utilization, the Prefill and Decode phases can be deployed in clusters with different specifications and architectures, forming a disaggregated framework. This architecture significantly boosts the throughput of LLM inference systems. For details, see Reason for PD Disaggregation.

In such a disaggregated framework, the KV cache generated during Prefill must be transmitted to the Decode nodes for incremental iterative inference. LLM-DataDist, a distributed cluster and data management component for LLMs, provides simple APIs for building this disaggregated architecture. As shown in the following figure, LLM-DataDist enables KV cache transmission and link management between Prefill and Decode nodes.