Graph Mode Entry

GE provides different interconnection solutions in different scenarios, facilitating the use of GE capabilities.

**Table 1** Graph mode entry
Entry	Interconnection Solution Provided by GE	Reference
Executing an onnx/pb model in graph mode	GE provides the ATC command line tool or C++ Parser API to convert the models (in .onnx/.pb/*.air format) exported from frameworks such as PyTorch and TensorFlow into offline models (in .om format) that adapt to Ascend AI Processor and then execute them in graph mode.	ONNX Model Conversion Using ATC Commands Using Parser APIs to Parse the Original Model into a Graph
Using the GE graph mode in AI frameworks such as PyTorch	Currently, mainstream AI frameworks such as PyTorch, TensorFlow, MindSpore, and PaddlePaddle can be connected to GE. You only need to migrate and adapt the training code before using the GE capability. This accelerates execution training or online inference tasks on Ascend AI Processor.	Connecting the PyTorch Framework to the GE Connecting the TensorFlow Framework to the GE Connecting the MindSpore Framework to the GE Connecting the PaddlePaddle Framework to the GE
Using the GE graph mode in AI frameworks such as PyTorch	For other third-party frameworks, GE provides unified C++ graph development APIs for you to complete adaptation.	Using Graph Development APIs to Construct a New Graph
Using Graph development APIs to construct a new network	GE provides unified C++ graph development APIs for you to customize graph structures and execute them in graph mode on Ascend AI Processor.	Using Graph Development APIs to Construct a New Graph

Connecting the PyTorch Framework to the GE

The PyTorch network can be connected to the GE through the TorchAir component of Ascend Extension for PyTorch. Torch Ascend Intermediate Representation (TorchAir) is an extended library that supports the graph mode in Ascend Extension for PyTorch (torch_npu). It connects to Dynamo of PyTorch to convert the FX (functionalization) graph of PyTorch into a computational graph indicated by Ascend intermediate representation (IR). GE is then used to optimize the build of the computational graph and deliver the graph to the Ascend hardware for execution.

Figure 1 Software architecture of the Ascend platform in PyTorch graph mode

The Ascend compilation backend provided by TorchAir can be passed to the torch.compile API of PyTorch as a parameter to enable the graph mode. An example is shown below:

      
           # Import torch_npu and then torchair.
import torch
import torch_npu
import torchair

# Define a model.
class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x, y):
        return torch.add(x, y)

# Instantiate the model.
model = Model()

# Obtain the default backend provided by the NPU from TorchAir.
config = torchair.CompilerConfig()
npu_backend = torchair.get_npu_backend(compiler_config=config)

# Use the backend of TorchAir to call the compile API for compiling the model.
opt_model = torch.compile(model, backend=npu_backend)

# Execute the compiled model.
x = torch.randn(2, 2)
y = torch.randn(2, 2)
opt_model(x, y)

The method is as follows:

Migrate the original PyTorch network to the Ascend platform. For details about the migration, see PyTorch Training Model Porting and Tuning Guide.
Use TorchAir to enable the graph model for inference. For details about the configuration method, see PyTorch Graph Mode User Guide (TorchAir).

Connecting the TensorFlow Framework to the GE

The TensorFlow 1.15 network is executed in graph mode. It can be connected to the GE through TF Adapter to convert the front-end TensorFlow model into a computational graph indicated by Ascend IR. GE is then used to optimize the build of the computational graph and deliver the graph to the Ascend hardware for execution. The TensorFlow 2.6.5 network is in eager mode (single-operator mode) by default. That is, each operator is executed and returned immediately after being delivered. TensorFlow 2.6.5 provides the tf.function decorator, which is used to encapsulate the TF2 operators called in Python functions into a graph for execution, achieving better performance benefits. For current TensorFlow 2.6.5, the Ascend platform supports executing only function operators decorated by tf.function on Ascend hardware with the accelerated mode, that is, the network can be executed only in graph mode.

Figure 2 Software architecture of the Ascend platform in TensorFlow graph mode

For details about how to migrate the original TensorFlow 1.15 network to the Ascend platform, see TensorFlow 1.15 Model Porting Guide.

For details about how to migrate the original TensorFlow 2.6.5 network to the Ascend platform, see TensorFlow 2.6.5 Model Porting Guide.

Connecting the MindSpore Framework to the GE

The MindSpore network natively supports the GE. You can directly use the GE capability based on the MindSpore framework. For details, visit https://www.mindspore.cn/.

Connecting the PaddlePaddle Framework to the GE

For details about the usage, see PaddlePaddle Documents.