Asynchronous scheduling is a scheduling algorithm that overlaps data processing latency with model inference time. It addresses the inefficiency of serial CPU and NPU usage in synchronous inference scenarios. In synchronous inference scenarios, the inference process can be divided into the following three phases based on execution on the CPU or NPU:
- Request scheduling and preparation phase (executed on the CPU)
- Model inference and sampling phase (executed on the NPU)
- Result judgment and response phase (executed on the CPU)
CPU and NPU tasks can be executed concurrently because they use different computing resources. To improve the overall resource utilization and throughput of the system, MindIE leverages the preceding characteristics and uses multiple threads to implement asynchronous scheduling.
It is worth noting that in this mode, requests that have entered the end of sequence (EOS) state will be calculated again, which may slightly waste NPU computing and memory resources.
Therefore, this feature is typically applicable to scenarios where [object Object] is large and the output length is long.
- This feature is supported in the prefill-decode co-location and prefill-decode disaggregation scenarios.
- This feature cannot be used with Look Ahead or Memory Decoding.
- Currently, this feature does not support postprocessing parameters related to multi-sequence inference, such as
[object Object],[object Object], and[object Object].
Set the following environment variable to enable asynchronous scheduling.
[object Object][object Object]
Open the
[object Object]file of the server.- Installation using the
[object Object]package:
[object Object]- Installation using the
[object Object]package:
[object Object]- Installation using the
Set serving parameters. For details about the serving parameters, see .
Start the service.
- Installation using the
[object Object]package:
[object Object]- Installation using the
[object Object]package:
[object Object]- Installation using the
(Optional) Use the AISBench tool to start tuning. For details about the AISBench tool, see "Auxiliary Tools" > "Performance/Accuracy Test Tool" in MindIE Motor Development Guide.