Introduction to DataFlow

DataFlow is an application development and execution framework for heterogeneous systems. It provides APIs to implement service logic. The framework hides hardware and network differences and improves application performance and throughput in asynchronous pipelines. DataFlow offers simple, flexible deployment options. You can easily adjust the number and location of service instances to optimize resource utilization and cut costs.

The following figure shows how DataFlow asynchronous pipelines improve throughput.

The sample application consists of three computing functions. Computing functions 1 and 3 are user-defined, while function 2 runs a computational graph. In the original workflow, the three computing functions run serially.

DataFlow uses FlowGraph to handle the entire computation process. Each computing function matches a FlowNode on the graph. The node function is carried by FuncProcessPoint or GraphProcessPoint. Nodes are connected through asynchronous data flows. With multiple data inputs, FlowGraph allows several FlowNodes to process different data simultaneously, turning sequential operations into asynchronous ones.

The following figure shows the relationship between DataFlow nodes.

• FlowGraph: a DataFlow directed acyclic graph (DAG), consisting of input node FlowData and compute node FlowNode. The data flows are directed and cyclic expressions are not allowed.
• FlowData: input node of FlowGraph. FlowData defines the name and type of an input node.
• FlowNode: compute node of FlowGraph. FlowNode defines the name, number of inputs, and number of outputs of a compute node. The following two types are supported:
  • FuncProcessPoint: compute ProcessPoint of functions, which implements user-defined functions through UDF.
  • GraphProcessPoint: compute ProcessPoint of graphs, which implements the compute logic of users through IR graph construction.

DataFlow allows users to write custom processing functions through FuncProcessPoint and GraphProcessPoint and run the functions in FlowModel mode through DataFlow graph construction.

For more information about DataFlow, see DataFlow Framework.