[object Object]

An SLO defines a target value for a specific metric over a certain period of time. To handle high-concurrency client requests and improve system throughput while meeting SLO requirements, the following approaches are provided:

1. Prefill/Decode phase selection algorithm based on TTFT/TPOT latency prediction and the Least Laxity First (LLF) algorithm. This algorithm collects TTFT and TPOT latency data for fitting modeling to predict the execution time of each prefill or decode phase, and uses the LLF algorithm to determine whether prefill or decode is executed for the next batch. It is suitable for scenarios with strict requirements on both TTFT and TPOT, enabling higher throughput under high-concurrency workloads while meeting SLO requirements.

2. Dynamic batch size adjustment algorithm based on real-time TPOT awareness. This algorithm continuously monitors the system TPOT latency and compares it with the SLO-defined decode latency target. Depending on the comparison result, [object Object] and [object Object] are dynamically adjusted to prevent all requests from being loaded into on-chip memory, which could cause system congestion and degrade throughput. This algorithm is suitable for scenarios with strict requirements on TPOT, prioritizing responses to requests already loaded into on-chip memory under high-concurrency workloads. Due to real-time fluctuations in TPOT data collection, the actual latency may deviate by roughly 10% from the configured target.

[object Object]

• Only the Atlas 800I A2 inference server supports this feature.
• DeepSeek-R1, DeepSeek-V3, and Qwen series models support this feature.
• This feature applies only to PD co-location and cannot be enabled together with the SplitFuse feature.
• This feature provides significant benefits for short outputs (less than 256 tokens). As the output length increases, the throughput gain decreases.

[object Object]

describes the parameters required for enabling the SLO scheduling tuning feature.

Table 1 Parameters of the SLO scheduling tuning feature[object Object][object Object]

[object Object]undefined[object Object]

This section describes how to use the SLO tuning optimization function.

1. Open the [object Object] file of MindIE Motor.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

2. Set serving parameters. Add the [object Object], [object Object], [object Object], and [object Object] fields (the following bold parts) to the [object Object] file of the server. For details about the parameter fields, see . For details about the serving parameters, see . The following is an example of parameter configuration.

   [object Object]

3. Start the service.

   • Installation using the [object Object] package:
   [object Object]
   • Installation using the [object Object] package:
   [object Object]

4. Start tuning. This example uses the AISBench tool and GSM8K dataset, with concurrency set to [object Object]. The AISBench tool is configured as follows. For details, see "" in Quick Start.

   [object Object]