[object Object]

The SplitFuse feature is used to split a long prompt request into smaller chunks and schedule the chunks in multiple forward steps. The prompt request is generated only after the last forward step is complete. Short prompt requests are combined to accurately fill the gap of steps. In this way, the calculation workload of each step is basically the same, which can achieve a more stable average latency of all requests.

When MindIE uses the prefill-decode co-location policy by default, requests in the prefill and decode phases are not combined into a single batch. However, with SplitFuse enabled, MindIE integrates prefill requests into the same batch when decode requests are processed first and the batch size is less than [object Object].

When [object Object] is greater than [object Object], SplitFuse splits it as follows:

• In each inference round: , where:
• In the prefill phase, [object Object] indicates the number of input tokens, and in the decode phase, each request has one token:

Two key behaviors:

1. Long prompts are split into smaller chunks and scheduled in multiple iterations. Only after the last iteration, tokens can be generated.

2. Short prompts may also be split into small chunks to ensure optimal computing efficiency.

Advantages:

• Faster response: The latency in processing long prompts is reduced, improving user experience.

• Efficiency improvement: Proper combination of short prompts ensures that a model runs at a high throughput.

• Enhanced consistency: Unified forward propagation can reduce latency fluctuation and stabilize generation frequency.

[object Object]

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• The Llama 3.1 70B floating-point model, Qwen2, Qwen2.5, and Qwen3 series models support this feature.
• This feature supports only the W8A8 quantization.
• This feature cannot be used with Multi-LoRA, Function Call, parallel decoding, MTP, or long sequence.
• This feature supports the [object Object], [object Object], and [object Object] postprocessing parameters.

[object Object]

and list the supplementary parameters required for enabling the SplitFuse feature.

Table 1 SplitFuse parameter: [object Object] in [object Object] [object Object][object Object]

[object Object]undefined

Table 2 SplitFuse parameter: [object Object] [object Object][object Object]

[object Object]undefined[object Object]

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add the [object Object] and [object Object] parameters to the [object Object] file of the server. For performance tuning, you need to edit [object Object] in the [object Object] configuration file. It is recommended that you configure the [object Object] parameter when a fixed chunk size is required. In other scenarios, you can use the default dynamic splitting configuration.

   For details about the SplitFuse parameters, see and . For details about the serving parameters, see . The following is an example of parameter configuration.

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

4. Use the AISBench tool to perform a performance test. For details, see "" in Quick Start.

5. Adjust parameters based on the actual values of the TTFT and decode latency.

   • If both the TTFT and decode latency (average value: P90) meet the threshold requirements, increase the value of [object Object].
   • If the average decode latency is less than the restricted threshold while the average TTFT is not, the value of [object Object] is greater than the system throughput. In this case, decrease the value of [object Object].
   • If the average TTFT and decode latency meet the threshold requirements but the average P90 decode latency does not, reduce the chunk size. However, this operation may affect the overall throughput.
   • When input questions vary in length, the prefill-decode co-location policy tends to generate more scheduling bubbles. In contrast, the SplitFuse feature is less impacted by such bubbles, resulting in superior performance.