GE Working Principles

Graph Construction

GE provides two graph construction methods:

• ONNX or PB model graph construction: You can use the ATC command line tool or C++ Parser API to map front-end framework operators to CANN operators one by one and parse framework model files (in formats such as *.onnx and *.pb) into computational graphs represented by Ascend IR.
  Figure 1 Parsing files into computational graphs using Parser APIs
  

  For details about the ATC command line tool, see ATC.

  For details about how to use Parser APIs to construct a graph, see Using Parser APIs to Parse the Original Model into a Graph.

• Using graph engine APIs to construct a new graph: You can use graph engine APIs to combine computational functions (operators) to construct a computational graph represented by Ascend IR. The following figure shows the basic process of constructing a graph. For more information, see Using Graph Engine APIs to Construct a New Graph.
  Figure 2 Constructing a new computational graph
  

Graph Build and Optimization

For computational graphs represented by Ascend IR, the GE performs a series of build and optimization and generates an offline OM model to adapt to the running requirements of the underlying hardware. The main process is as follows:

• Graph preparation: Based on the input tensor information, operator logic, and operator attributes, the output tensor description of the operator is inferred in advance, including the tensor shape, data type, and data layout format. In the operator graph construction preparation phase, memory can be statically allocated to all tensors to avoid the overhead caused by dynamic memory allocation. This process is called inferShapeAndType and inferFormat. In addition, algorithm-level optimizations that are irrelevant to hardware are performed, including but not limited to constant folding and redundant branch elimination.
• Graph partitioning: Operators are classified based on the execution engine (such as AI Core or AI CPU) and partitioned into different subgraphs to facilitate subsequent optimization on different hardware.
• Graph optimization: Graph optimization methods such as operator fusion are used to improve graph run performance. Hardware-irrelevant optimizations can be performed. For example, multiple operators can be fused into one or more operators to save the computation time. Hardware-related optimizations can also be performed. For example, UB fusion can be used to shorten the data movement time in the hardware memory, thereby improving execution efficiency.

  The Atlas 350 Accelerator Card does not support UB fusion.

• Graph build: Running resources, including memory and stream resources, are allocated based on the computational graph, and an .om offline model is compiled and generated.

Graph Loading and Run

Load the generated offline model file, allocate running resources, and deliver streams/tasks to the device for execution. The main process is as follows:

• Graph loading: The offline model is parsed, memory resources are allocated, and stream running resources are created.
• Graph run: The input data is copied, and the stream/task is delivered to the device. The AI Core or AI CPU executes the corresponding operator. Once the device finishes the computation, it returns the result to the host user program.