[object Object]

In LLM inference scenarios, conventional auto-regressive decoding is inherently slow due to its step-by-step nature, which restricts concurrency. Although the inference phase is constrained by memory bandwidth, it often has excessive computing resources. To address this imbalance, parallel decoding introduces speculative execution—an optimization technique commonly used in processor architectures—that leverages the excessive computing resources to improve concurrency. However, enabling parallel decoding requires the prompt input to retain a trie-tree and a draft token map, which affects the TTFT.

Advantages of parallel decoding:

In small-batch inference scenarios—such as those involving sufficiently long inputs/outputs or code generation—parallel decoding can offset limited memory bandwidth by utilizing excess computing resources, thereby enhancing computing efficiency. The effectiveness of parallel decoding is closely tied to the ratio of validated tokens. As a result, greedy decoding offers the greatest benefit, while sampling and penalty mechanisms may reduce its impact.

To fully leverage parallel decoding, the following conditions should be met:

1. A low number of concurrent requests, constrained memory bandwidth, and surplus computational resources.
2. Sufficiently long input to provide an initial source of candidate tokens.
3. Extended output length, allowing parallel decoding to reduce inference steps and deliver performance gains.

Two parallel decoding algorithms are supported, distinguished by their respective methods of candidate token generation, as shown in .

Table 1 Parallel decoding algorithms[object Object][object Object]

[object Object]undefined[object Object]

• The Atlas 800I A2 inference server and Atlas 300I Duo inference card support this feature.
• Only the Llama3 series, Qwen2 series, Qwen2.5 series, Qwen3-14B, and Qwen3-32B models support this feature.
• Parallel decoding supports only W8A8 quantization and sparse quantization.
• This feature cannot be used with prefill-decode disaggregation, Multi-LoRA, SplitFuse, long sequence, MTP, asynchronous scheduling, or multi-server inference.
• Currently, this feature does not support postprocessing parameters related to multi-sequence inference, such as [object Object], [object Object], [object Object], [object Object], and [object Object].
• Streaming inference is not supported in parallel decoding scenarios.
• Parallel decoding penalty postprocessing supports only repetition penalty.
• Health check is not supported in parallel decoding scenarios.
• The lookahead and memory_decoding algorithms cannot be enabled at the same time.

[object Object]

to describe the parameters required for enabling the parallel decoding feature.

Table 2 memory_decoding parameter: [object Object] in `ModelDeployConfig [object Object][object Object]

[object Object]undefined

Table 3 memory_decoding parameter: [object Object]

[object Object]undefined

Table 4 memory_decoding parameter: [object Object]

[object Object]undefined

Table 5 Lookahead parameter: [object Object] in [object Object]

[object Object]undefined

Table 6 Lookahead parameter: [object Object] [object Object][object Object]

[object Object]undefined[object Object]

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add parameters to the [object Object] file of the server according to to . For details about the serving parameters, see . The following is an example of parameter configuration.

   Example of parallel decoding configuration for the memory_decoding algorithm:

   [object Object]

   Configuration example of the lookahead algorithm:

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]