Reason for PD Disaggregation

In practice, LLM inference consists of a Prefill phase followed by multiple iterative Decode steps. A prerequisite for enabling continuous batching during the Decode phase is that each scheduled request has sufficient idle resources to complete its Prefill computation. In conventional co-located deployments, where Prefill and Decode share the same compute resources, incoming Prefill requests are processed with priority. This can disrupt ongoing Decode execution, compromise the stability of token between tokens (TBT), and lead to inconsistent latency. As illustrated below, when request 5 or request 6 arrives, the system may prioritize their Prefill phases, delaying requests 2, 3, and 4 and causing TBT fluctuations.

Given the fundamentally distinct computation and communication characteristics of the Prefill and Decode phases, co-locating them imposes fundamental limitations on both performance and resource efficiency. To address this, PD disaggregation deploys the two phases on separate clusters with specifications and architectures tailored to their respective requirements. Paired with a service-layer task scheduler, this approach maximizes Decode-phase batch concurrency through continuous batching while maintaining target TTFT and TBT performance, thereby improving overall throughput and resource utilization.

With PD disaggregation, Prefill and Decode execution no longer interfere with each other. As shown in the figure, this enables the system to provide users with a consistently stable TBT.