Terminology

Prefill phase

Also called Prompt or full inference phase, where the input from the user is inferred for the first time. The input in this phase is the complete sentence entered by the user.

Decode phase

Also called incremental inference phase, where iterative inference is performed based on the output of the Prefill phase. In this phase, the input of each iteration is the output of the previous inference.

PD disaggregation

The Prefill phase and Decode phase are separated.

KV Cache

KV cache is a technology used in LLMs to avoid repeated context computation in each generation step by caching computed Key and Value. In this document, it refers to the cache of computed keys and values.

PagedAttention

PA for short. It is a function of requesting the KV cache in discrete block mode, which saves the memory used by the KV cache.

block_table

A set of block indices occupied by the requested KV cache in the PagedAttention scenario.

Request

Process when a question raised by a user is answered after going through the Prefill and Decode phases.

role

Service role represented by LLM-DataDist. The service role can be Prefill or Decoder.

cluster_id

ID of the cluster where LLM-DataDist is located. The Decode side uses the value of cluster_id to find the corresponding link that leads to the KV cache of the corresponding request on the Prefill side. When different shards of a model are deployed on different devices and PD equal sharding is enabled, cluster_id must be set to the same value for all shards of the split model.

Dynamic cluster scaling

The number of clusters and the ratio of Prefill to Decode clusters are dynamically adjusted based on service load to save resources during off-peak hours and improve throughput during peak hours.

D2D transmission

Data is transmitted from a device to another device.

D2H transmission

Data is transmitted from a device to a host.

H2D transmission

Data is transmitted from a host to a device.

D2RH

Data is transmitted from a device to a remote host.

RH2D

Data is transmitted from a remote host to a device.

TTFT

Short for "Time To First Token."

A performance metric in LLM inference that measures the latency from input submission to the output of the first token.

It refers to the time taken for a batch of requests to complete the Prefill phase after entering the inference system, and represents the system's response time to generate the first character.

TBT

Short for "Time Between Tokens."

Latency between tokens, that is, the time taken for each decoding step. It reflects the interval between generating consecutive tokens during online LLM processing, indicating the smoothness of the generation process.

Continuous batching

A technique for optimizing computational efficiency in LLM inference. It processes multiple inference requests as a batch to fully utilize compute resources, thereby improving inference speed and throughput. The core idea is to minimize idle time and maximize resource utilization.

Unilateral link establishment

The client initiates a link establishment request to the server.

Bilateral link establishment

All LLM-DataDist instances can initiate link establishment with each other simultaneously.