[object Object]

Currently, the KV cache mechanism is widely used in LLM inference systems. However, this mechanism has the following problems:

1. As sequence lengths supported by LLMs continuously increase, the graphics memory resources required by the KV cache also sharply increase.
2. The KV cache is valid only for the current session. If duplicate token sequences exist across sessions, the KV cache cannot be reused.

Prefix cache stores the KV cache of completed sessions in a hash table. For a new session request, the system checks whether the same token sequence exists in the table. If yes, the previously computed KV cache can be reused across sessions.

Advantages:

• Shorter prefill time: By reusing the KV cache corresponding to the repeated cross-session token sequence, the computation time for some prefix tokens can be reduced, thereby decreasing the prefill time.
• More efficient graphics memory usage: When the sessions being processed have a common prefix, the KV cache of the common prefix can be shared, reducing redundant graphics memory usage.

[object Object]

• The Atlas 800I A2 inference server, Atlas 300I Duo inference card, and Atlas 800I A3 SuperPoD server support this feature.
• The Qwen2 series, Qwen2.5 series, Qwen3 series, DeepSeek-R1, and DeepSeek-V3/V3.1 models support this feature.
• The KV cache of the public prefix tokens is reused only when the number of cross-session public prefix tokens is greater than or equal to the block size.
• Prefix cache supports only W4A8 quantization, W8A8 quantization, PDMIX quantization, and sparse quantization.
• This feature cannot be used with Multi-LoRA.
• This feature can be used with prefill-decode disaggregation, parallel decoding, MTP, KV cache pooling, asynchronous scheduling, SplitFuse, context parallel + sequence parallel, and C8 quantization.
• This feature supports the [object Object], [object Object], and [object Object] postprocessing parameters.
• In the prefill-decode disaggregation scenario, this feature needs to be enabled only on the prefill node.
• You are advised not to enable this feature when the prefix reuse rate is low or no prefix is reused.
• The combination of prefix cache, context parallel, sequence parallel, and function call (multiturn) is not supported.

[object Object]

, , and list the supplementary parameters required for enabling the Prefix Cache feature.

Table 1 Prefix Cache parameter (1): [object Object] in [object Object] [object Object][object Object]

[object Object]undefined

Table 2 Prefix Cache parameter (2): [object Object] [object Object][object Object]

[object Object]undefined

Table 3 Prefix Cache parameters (3): [object Object] in [object Object] [object Object][object Object]

[object Object]undefined[object Object]

The following uses multi-round dialog as an example to describe how to use the prefix cache feature.

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add parameters to the [object Object] file of the server according to to . For details about the serving parameters, see . The following is an example of parameter settings.

   The following uses the DeepSeek-R1 model as an example to describe how to enable the prefix cache feature.

   [object Object]

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

4. Send a request for the first time. The prompt is the first round of questions.

   [object Object]

5. Send a second request with the prompt in the format of "first-round question + first-round answer + second-round question". In this case, the first-round question is a reusable public prefix. (The actual reused part may not be the complete prompt of the first-round question. The cache is implemented in the unit of block, and the prefix cache is stored in a multiple of [object Object]. For example, if the number of tokens in the first-round question is 164, and [object Object] is [object Object], only the first 128 tokens are reused.)

   [object Object]