LLM Inference Process

The launch of ChatGPT marked the advent of interactive AI as a commercially mature technology, significantly accelerating research and development in LLMs. Today, the most widely adopted LLMs include the GPT series, Llama series, and GLM series. All are built upon the Transformer architecture, predominantly employing a Decode-Only design. During inference, these Transformer-based Decode-Only LLMs operate in auto-regressive generation (AR) mode: each token is generated through a forward pass of the LLM, which involves computation across all N Transformer layers. Consequently, generating a complete sequence of M tokens requires M full forward passes.

To mitigate the computational workload during the Decode phase, KV caching has become an indispensable technique in modern LLM inference systems. With KV caching, the inference process is typically divided into two distinct phases: Prefill and Decode.

• Prefill phase: The user prompt is fed into the model, which computes intermediate results, writes them to the KV cache, and generates the first token. As the prompt sequence length increases linearly, services with strict Time To First Token (TTFT) requirements often execute LLM inference in single-batch mode during this phase. This phase is compute-bound.
• Decode phase: The previously generated token is fed back into the model, which retrieves the KV cache generated earlier from the memory for computation. With KV caching, the per-token computation overhead is minimal, and the primary bottleneck shifts to the memory bandwidth required to transfer model parameters. Therefore, the multi-batch mode is typically used to improve utilization. This phase is memory-bound.