[object Object]

MTP is a parallel decoding method used by DeepSeek to generate multiple tokens at a time. The core idea of MTP is that a model forecasts not just the subsequent token but several tokens concurrently during inference, which markedly enhances generation efficiency.

[object Object]

describes the parameters required for enabling the MTP feature.

Table 1 MTP parameter in [object Object] of [object Object] [object Object][object Object]

[object Object]undefined[object Object]

MTP can be used together with the following features:

1. Prefix cache and KV cache pooling
2. Asynchronous scheduling
3. KV cache int8 quantization
4. function call
5. Thinking analysis
6. Prefill-decode disaggregation (Both P and D nodes must be configured.)

MTP can be used together with the following features in some scenarios:

1. context_parallel
2. sequence_parallel

[object Object]

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• Only the W8A8 and KV cache INT8 quantization models of DeepSeek-R1 and DeepSeek-V3 support this feature.
• This feature supports W4A8 quantization.
• This feature cannot be used with parallel decoding, Multi-LoRA, or SplitFuse.
• When this feature is used together with context_parallel and sequence_parallel in the PD co-location scenario, num_speculative_tokens can only be set to 1.
• When context_parallel and sequence_parallel are used together with the MoE EP, only the P node supports context_parallel and sequence_parallel.
• This feature does not support postprocessing parameters related to multi-sequence inference, such as [object Object], [object Object], [object Object], and [object Object].
• MTP postprocessing supports only repetition penalty.

[object Object]

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add the [object Object] field to the [object Object] file of the server. For details about the parameter fields, see . For details about the serving parameters, see . The following is an example of parameter configuration.

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]