AutoFuse Overview

Introduction

AutoFuse is an Ascend C-based automatic fusion framework. It supports automatic fusion scope identification, automatic operator code generation, Auto Tiling optimization, dynamic shape, and mixed accuracy. On an algorithm network, there is a large number of vector computations, causing frequent memory transfers. This causes memory bound issues. AutoFuse helps automatically fuse multiple operators into one, thereby reducing the number of operators and memory transfers on the network, reducing memory bound issues, releasing Ascend computing power, and improving model execution performance.

Figure 1 shows the benefit principle. As automatic fusion combines multiple operators into a single operator, it brings benefits in Memory Transfer Engine (MTE) movement and dynamic shape scheduling overheads. For networks with small shapes and MTE bound, it usually yields positive returns.

Automatic fusion is supported by the following AI processors:

• Atlas 350 Accelerator Card
• Atlas A3 training product / Atlas A3 inference product
• Atlas A2 training product / Atlas A2 inference product

Technical Roadmap

The AutoFuse solution provides two fusion implementation roadmaps based on the underlying unified Ascend C IR schedule and code generation capabilities of the Ascend NPU. Ascend C IR (referred to as AscIR) is the IR for modeling in the Ascend C programming language. Figure 2 shows the roadmaps.

• Roadmap 1 is developed based on the Ascend GE framework and focuses on the NPU affinity. The three core capabilities are as follows:
  • Symbolic shape inference of Ascend Intermediate Representation (Ascend IR): Variable symbols are used to express the dynamically changing shapes, so that code is generated based on the symbolic shape during compilation.
  • Lowering from Ascend IR to AscIR: The low-level AscIR is used to express the computation logic of Ascend IR and determine the fusion structure.
  • Fusion policy: Based on the characteristics and constraints of AscIR, the loop axes between fusion structures are combined to obtain the optimal fusion solution.

• Roadmap 2: The solution connects to the PyTorch Inductor, focuses on the ecosystem support, reuses the fusion capability of the Inductor, and converts the fusion structure expressed by Inductor IR into an Ascend C IR graph to generate code. This technical roadmap is not supported in the current version.

Fusion Principle

Automatic fusion consists of two parts: automatic fusion scope determination, and automatic generation of fused kernel source code and binary based on the fusion scope. The former is called the automatic fusion frontend, and the latter is called the automatic fusion backend (corresponding to the common underlying capability in Figure 2).

The frontend determines which operators can be fused based on certain rules or configurations, and determines the fusion scope of a fused operator. The fusion scope is expressed by FusedGraph, as shown in Figure 3. FusedGraph contains at least one AscBackend node. AscBackend can be considered as an Ascend IR operator similar to ge::op::partitionedcall. It carries a subgraph object. An AscBackend node carries an AscGraph attribute, and an AscGraph contains multiple AscIR nodes. For details about AscIR and AscGraph, see AscIR and AscGraph.

The backend consists of the Schedule, Codegen, and Auto Tiling modules. After receiving the FusedGraph, the Schedule module generates the corresponding TilingGroups for the compute and movement nodes, and constructs the normalized TilingGroup applicable to the entire fused operator based on the TilingGroup fusion policy. Finally, the tiling policies are generated for the FusedGraph based on the normalized TilingGroup. After Codegen implementation and Auto Tiling, the kernel source code, tiling_func source code, and tiling_data source code are generated, and the host and device deliverables are compiled. For details about how automatic fusion takes effect, see AutoFuse Enabling Method. For details about the implementation principles of Schedule, Codegen, and Auto Tiling, see the corresponding sections.