[object Object]

Micro-batch processing is a technique where data is split into smaller batches for execution. In the current implementation, an additional data stream is created to split a batch of data into two batches, which are executed on two separate data streams. During computation in Dataflow 1, Dataflow 2 can communicate, overlapping computation and communication overhead. This fully utilizes hardware resources, improving inference throughput.

Figure 1 Micro-batch processing with dual data streams

Data streams are synchronized using the event mechanism, ensuring that computation and communication tasks do not conflict with each other and preventing hardware resource contention. This feature is typically used in the prefill phase because communication operators consume long execution time and the execution duration of communication and compute operators is balanced. In this implementation, the overlap between computation and communication exceeds 70%.

[object Object]

• This feature is disabled by default.
• This feature cannot be enabled together with the merged communication-computing operator feature.
• Only the Qwen2.5 series, Qwen3 dense series, DeepSeek-R1, and DeepSeek-V3.1 models support this feature.
  • For the Qwen model, this feature can be enabled together with parallel decoding, asynchronous scheduling, SplitFuse, and prefix cache.
  • For the Deepseek model, this feature can be enabled together with the MTP feature.
• Enabling this feature will occupy extra graphics memory. In serving scenarios, if the number of KV caches decreases, scheduling will be affected and the throughput will decrease. Therefore, you are advised not to enable this feature when the graphics memory is limited.

[object Object]

describes the parameters required for enabling the micro batch feature.

Table 1 Micro batch parameters: [object Object] in [object Object] [object Object][object Object]

[object Object]undefined[object Object]

1. Open the [object Object] file of the server.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add the [object Object] field (the following bold part) to the [object Object] file of MindIE. For details about the parameter fields, see . For details about the serving parameters, see . The following is an example of parameter configuration.

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]