SplitFuse

The SplitFuse feature is used to split a long prompt request into smaller chunks and schedule the chunks in multiple forward steps. The prompt request is generated only after the last forward step is complete. Short prompt requests are combined to accurately fill the gap of steps. In this way, the calculation workload of each step is basically the same, which can achieve a more stable average latency of all requests.

When MindIE uses the prefill-decode co-location policy by default, requests in the prefill and decode phases are not combined into a single batch. However, with SplitFuse enabled, MindIE integrates prefill requests into the same batch when decode requests are processed first and the batch size is less than maxBatchSize.

When feedforward is greater than splitChunk tokens, SplitFuse splits it as follows:

• In each inference round: , where:
• In the prefill phase, tokens indicates the number of input tokens, and in the decode phase, each request has one token:

Two key behaviors:

1. Long prompts are split into smaller chunks and scheduled in multiple iterations. Only after the last iteration, tokens can be generated.

2. Short prompts may also be split into small chunks to ensure optimal computing efficiency.

Advantages:

• Faster response: The latency in processing long prompts is reduced, improving user experience.

• Efficiency improvement: Proper combination of short prompts ensures that a model runs at a high throughput.

• Enhanced consistency: Unified forward propagation can reduce latency fluctuation and stabilize generation frequency.

Constraints

• The Atlas 800I A2 inference server and Atlas 800I A3 SuperPoD server support this feature.
• The Llama 3.1 70B floating-point model, Qwen2, Qwen2.5, and Qwen3 series models support this feature.
• This feature supports only the W8A8 quantization.
• This feature cannot be used with Multi-LoRA, Function Call, parallel decoding, MTP, or long sequence.
• This feature supports the n, best_of, and use_beam_search postprocessing parameters.

Parameter Description

Table 1 and Table 2 list the supplementary parameters required for enabling the SplitFuse feature.

Table 1 SplitFuse parameter: ModelConfig in ModelDeployConfig

Table 2 SplitFuse parameter: ScheduleConfig

Inference

1. Open the config.json file of the server.

   • Installation using the .whl package:

   cd {MindIE_installation_directory}/mindie_llm/
   vi conf/config.json

   • Installation using the .run package:

   cd {MindIE_installation_directory}/latest/mindie-service
   vi conf/config.json

2. Set serving parameters. Add the plugin_params and templateType parameters to the config.json file of the server. For performance tuning, you need to edit ScheduleConfig in the config.json configuration file. It is recommended that you configure the prefillChunkSize parameter when a fixed chunk size is required. In other scenarios, you can use the default dynamic splitting configuration.

   For details about the SplitFuse parameters, see Table 1 and Table 2. For details about the serving parameters, see Configuration Parameters (Serving). The following is an example of parameter configuration.

           "ModelDeployConfig":
           {
               "maxSeqLen" : 65536,
               "maxInputTokenLen" : 65536,
               "truncation" : 0,
               "ModelConfig" : [
                   {
                       "modelInstanceType": "Standard",
                       "modelName" : "llama3-70b",
                       "modelWeightPath" : "/home/models/llama3-70b/",
                       "worldSize" : 8,
                       "cpuMemSize" : 5,
                       "npuMemSize" : -1,
                       "backendType": "atb",
                       "plugin_params": "{\"plugin_type\":\"splitfuse\"}"
                   }
               ]
           },
           "ScheduleConfig":
           {
               "templateType": "Mix",
               "templateName" : "Standard_LLM",
               "cacheBlockSize" : 128,
   
               "maxPrefillBatchSize" : 40,
               "maxPrefillTokens" : 65536,
               "prefillTimeMsPerReq" : 600,
               "prefillPolicyType" : 0,
   
               "decodeTimeMsPerReq" : 50,
               "decodePolicyType" : 0,
               "maxBatchSize" : 256,
               "maxIterTimes" : 512,
               "maxPreemptCount" : 0,
               "supportSelectBatch" : false,
               "maxQueueDelayMicroseconds" : 5000,
   
               "prefillChunkSize" : 1024,
               "maxNumPartialPrefills" : 64,
               "longPrefillTokenThreshold" : 1024,
               "maxLongPartialPrefills" : 8
           }

3. Start the service.

   • Installation using the .whl package:

   mindie_llm_server

   • Installation using the .run package:

   ./bin/mindieservice_daemon

4. Use the AISBench tool to perform a performance test. For details, see "Performance Test" in Quick Start.

5. Adjust parameters based on the actual values of the TTFT and decode latency.

   • If both the TTFT and decode latency (average value: P90) meet the threshold requirements, increase the value of RequestRate.
   • If the average decode latency is less than the restricted threshold while the average TTFT is not, the value of RequestRate is greater than the system throughput. In this case, decrease the value of RequestRate.
   • If the average TTFT and decode latency meet the threshold requirements but the average P90 decode latency does not, reduce the chunk size. However, this operation may affect the overall throughput.
   • When input questions vary in length, the prefill-decode co-location policy tends to generate more scheduling bubbles. In contrast, the SplitFuse feature is less impacted by such bubbles, resulting in superior performance.