Introduction to LLM-DataDist

In LLM inference scenarios, as the batch size increases, performance in the Prefill phase degrades linearly, while the Decode phase consumes significantly more memory. These two phases have different resource requirements; deploying them together leads to uneven resource allocation and high costs. The disaggregated inference architecture built with LLM-DataDist effectively addresses this issue by placing Prefill and Decode in clusters with different specifications and architectures, thereby improving performance, resource utilization, and overall system throughput.

LLM-DataDist is a distributed cluster and data management component for LLMs. It provides high-performance, zero-copy point-to-point data transmission capabilities through simple APIs. Leveraging the diverse communication links (RoCE, HCCS, and UB) of Ascend clusters, LLM-DataDist enables efficient KV cache transmission across instances and clusters. It can be integrated with mainstream LLM inference frameworks such as vLLM and used to build distributed data management systems. LLM-DataDist offers two main functions: link management and cache management.

• Link management establishes and tears down links between clusters, enabling dynamic scaling.
• Cache management manages KV cache and provides point-to-point transmission between the Prefill (P) and Decode (D) sides.