Documentation

Overview

Graph mode is a run mode of a neural network model. In this mode, the model computation process is constructed into a graph, which is then delivered through GE to the Ascend hardware for execution. Compared with the eager mode, the graph mode can accelerate model execution and reduce model memory usage through GE technologies such as computational graph optimization, multi-stream parallelism, memory reuse, and model offload.

The following figure shows the process of integrating operators into a graph. After an operator project is created, define the operator prototype based on the project code framework, implement the operators on the kernel, implement tiling on the host, and integrate the operators into the graph. Use the project build script to build and deploy the operators. Then, execute the operators based on the graph IR, for example, call custom operators through single-operator model execution or IR graph construction. Follow the development process based on Project-based Operator Development. In addition to the operator implementation file described in Project-based Operator Development, the code file for integrating operators to the graph also needs to be delivered.

  1. Set up the environment.
    1. Install the CANN software. For details, see Environment Setup.
    2. Create an operator project. Use msOpGen to create an operator development project.
  2. Implement operators.
    • Define the operator prototype. The operator prototype describes the input, output, and attributes of an operator as well as the implementation information of the operator on the AI processor, and associates the operator with the function such as tiling implementation.
    • Implement operators on the kernel and implement tiling on the host. For details, see Operator Implementation. In project-based operator development, you can call tiling APIs to perform tiling development based on the programming framework provided by CANN, and you can also call corresponding APIs on the kernel to obtain tiling parameters. For details, see Operator Implementation on the Kernel and Tiling Implementation on the Host. The additional restrictions are also described in the preceding sections.
  3. Integrate operators into a GE graph. In this scenario, the implementation of adaptation functions such as shape inference needs to be provided.
  4. Build and deploy the operators. Use the project build script to build and deploy the operators.
  5. Build and run the graph: Execute operators based on graph IR, for example, call custom operators through single-operator model execution or IR graph constructionIR graph construction.