Parallel Decoding

In LLM inference scenarios, conventional auto-regressive decoding is inherently slow due to its step-by-step nature, which restricts concurrency. Although the inference phase is constrained by memory bandwidth, it often has excessive computing resources. To address this imbalance, parallel decoding introduces speculative execution—an optimization technique commonly used in processor architectures—that leverages the excessive computing resources to improve concurrency. However, enabling parallel decoding requires the prompt input to retain a trie-tree and a draft token map, which affects the TTFT.

Advantages of parallel decoding:

In small-batch inference scenarios—such as those involving sufficiently long inputs/outputs or code generation—parallel decoding can offset limited memory bandwidth by utilizing excess computing resources, thereby enhancing computing efficiency. The effectiveness of parallel decoding is closely tied to the ratio of validated tokens. As a result, greedy decoding offers the greatest benefit, while sampling and penalty mechanisms may reduce its impact.

To fully leverage parallel decoding, the following conditions should be met:

1. A low number of concurrent requests, constrained memory bandwidth, and surplus computational resources.
2. Sufficiently long input to provide an initial source of candidate tokens.
3. Extended output length, allowing parallel decoding to reduce inference steps and deliver performance gains.

Two parallel decoding algorithms are supported, distinguished by their respective methods of candidate token generation, as shown in Table 1.

Table 1 Parallel decoding algorithms

Constraints

• The Atlas 800I A2 inference server and Atlas 300I Duo inference card support this feature.
• Only the Llama3 series, Qwen2 series, Qwen2.5 series, Qwen3-14B, and Qwen3-32B models support this feature.
• Parallel decoding supports only W8A8 quantization and sparse quantization.
• This feature cannot be used with prefill-decode disaggregation, Multi-LoRA, SplitFuse, long sequence, MTP, asynchronous scheduling, or multi-server inference.
• Currently, this feature does not support postprocessing parameters related to multi-sequence inference, such as n, best_of, use_beam_search, logprobs, and top_logprobs.
• Streaming inference is not supported in parallel decoding scenarios.
• Parallel decoding penalty postprocessing supports only repetition penalty.
• Health check is not supported in parallel decoding scenarios.
• The lookahead and memory_decoding algorithms cannot be enabled at the same time.

Parameter Description

Table 2 to Table 6 describe the parameters required for enabling the parallel decoding feature.

Table 2 memory_decoding parameter: ModelConfig in `ModelDeployConfig

Table 3 memory_decoding parameter: ModelDeployConfig

Table 4 memory_decoding parameter: ScheduleConfig

Table 5 Lookahead parameter: ModelConfig in ModelDeployConfig

Table 6 Lookahead parameter: ModelDeployConfig

Inference

1. Open the config.json file of the server.

   • Installation using the .whl package:

   cd {MindIE_installation_directory}/mindie_llm/
   vi conf/config.json

   • Installation using the .run package:

   cd {MindIE_installation_directory}/latest/mindie-service
   vi conf/config.json

2. Set serving parameters. Add parameters to the config.json file of the server according to Table 2 to Table 6. For details about the serving parameters, see Configuration Parameters (Serving). The following is an example of parameter configuration.

   Example of parallel decoding configuration for the memory_decoding algorithm:

   "ModelDeployConfig" :
   {
       "maxSeqLen" : 2560,
       "maxInputTokenLen" : 2048,
       "truncation" : 0,
       "speculationGamma": 16,
       "ModelConfig" : [
           {
               "plugin_params":"{\"plugin_type\":\"memory_decoding\",\"decoding_length\":16,\"dynamic_algo\":true}",
               "modelInstanceType" : "Standard",
               "modelName" : "llama3-70b",
               "modelWeightPath" : "/data/weights/llama3-70b",
               "worldSize" : 4,
               "cpuMemSize" : 5,
               "npuMemSize" : -1,
               "backendType" : "atb",
               "trustRemoteCode" : false
           }
       ]
   }

   Configuration example of the lookahead algorithm:

   "ModelDeployConfig" :
   {
       "maxSeqLen" : 2560,
       "maxInputTokenLen" : 2048,
       "truncation" : 0,
       "speculationGamma": 30,
       "ModelConfig" : [
           {
               "plugin_params":"{\"plugin_type\":\"la\",\"level\":4,\"window\":5,\"guess_set_size\":5}",
               "modelInstanceType" : "Standard",
               "modelName" : "Qwen2.5-7B-Instruct",
               "modelWeightPath" : "/data/weights/Qwen2.5-7B-Instruct",
               "worldSize" : 1,
               "cpuMemSize" : 5,
               "npuMemSize" : -1,
               "backendType" : "atb",
               "trustRemoteCode" : false
           }
       ]
   }

3. Start the service.

   • Installation using the .whl package:

   mindie_llm_server

   • Installation using the .run package:

   ./bin/mindieservice_daemon