Prefill-Decode Disaggregation Architecture

In this architecture, the two phases (prefill and decode) of inference are deployed on different instances for computing, reducing mutual interference between prefill and decode phases and improving inference performance. The following figure shows its basic process.

Figure 1 Basic process of the prefill-decode disaggregation architecture

The prefill-decode disaggregation is a cluster-based architecture, with three necessary elements:

Coordinator
Prefill instance (P)
Decode instance (D)

The coordinator is responsible for releasing inference APIs, and prefill and decode instances are responsible for computing in their respective inference phases. The coordinator is implemented by the cluster management component, and the prefill and decode instances are implemented by the Server.

Based on this architecture, the Server does not directly provide APIs for end users. This section describes APIs provided by the Server for the cluster management component.

Parent topic: Prefill-Decode Disaggregation APIs