PyTorch-Based Online Model Inference

DataFlow can train models or perform online inference based on multiple deep learning frameworks to improve model throughput. For example, PyTorch-based online model inference steps include preprocessing, model inference, and postprocessing. The model inference step can be completed by multiple sub-models that are executed in serial mode based on the model structure.

According to the preceding service logic, DataFlow can encapsulate preprocessing, inference by multiple sub-models, and postprocessing into multiple FuncProcessPoints that can be executed asynchronously, and connect the data flows between them through an asynchronous queue, as shown in the following figure.

By converting the original serial execution process into a FlowGraph, asynchronous pipelines between different modules can be enabled during multi-round inference, improving the overall throughput of applications. For details about the sample code, see Sample Reference.