Application Scenarios of the Unilateral Communication Library

The unilateral communication library enables a disaggregated PD architecture for LLM inference.

In LLM inference, the user prompt is fed into the model during the Prefill phase, where the model computes intermediate results, writes them to the KV cache, and generates the first token. In the Decode phase, this first token is fed back into the model, which reads the previously generated KV cache from the memory to continue computation. For more details on KV cache-based LLM inference, see LLM Inference Process.

To improve performance and resource utilization, the Prefill and Decode phases can be deployed in clusters with different specifications and architectures, forming a disaggregated framework. This disaggregated framework significantly boosts the throughput of LLM inference systems. For details about why PD disaggregation is required, see Reason for PD Disaggregation.

In such a disaggregated PD framework, the KV cache generated during Prefill must be transmitted to the Decode nodes for incremental iterative inference. As a distributed cluster and data management component for LLMs, the unilateral communication library provides simple APIs for building this disaggregated architecture. As shown in the following figure, the unilateral communication library enables KV cache transmission and link management between Prefill and Decode nodes.