Fusion Condition Check
To determine whether two nodes can be fused, you need to consider the check results of both the CanFuse framework and Backend. The two nodes can be fused only if both results indicate that the fusion conditions are met. This section describes how they check fusion conditions.
Fusion Conditions Checked by the CanFuse Framework
- Memory read and write times can be reduced.
The nodes to be fused must share memory. As shown in the following figure, output 1 of node 1 is the input of node 3, and so they can be fused. The inputs of nodes 3 and 4 come from output 1 of node 1, and so they can be fused. After fusion, memory transfer times can be reduced. Although the inputs of nodes 2 and 3 come from node 1, they are not the same output of node 1. After fusion, data needs to be transferred twice, and memory transfer times cannot be reduced. Therefore, they cannot be fused.
Figure 1 Fusion condition 1 checked by CanFuse
- No loop is formed.
As shown in the following figure, a loop will be formed if nodes 1 and 3 are fused. Therefore, the nodes cannot be fused.
Figure 2 Fusion condition 2 checked by CanFuse
- The maximum number of fusion times is not exceeded. The default maximum number of fusion times is 64.
Fusion scale control prevents the usage of backend resources from exceeding the threshold. The number of nodes is controlled based on the number of nodes in the AscGraph generated by lowering. As shown in the following figure, if nodes 1 and 2 are fused, the total number of nodes is 9, and the nodes can be fused. If the total number of nodes exceeds the threshold after the fusion, they cannot be fused.
Figure 3 Fusion condition 3 checked by CanFuse
- The peak memory usage does not increase.
Over-fusion may increase the peak memory usage. Therefore, a balance between the execution performance and memory usage needs to be achieved. A complete peak memory evaluation is a complex process. Therefore, a simple policy is used for evaluation. If the node span (ID difference in topology sorting) exceeds the specified threshold, the node will not be fused. If a node can be horizontally fused, the system checks whether the output memory of the node will exceed 13 GB after fusion. If it will, the node is not fused.
As shown in the following figure, memory resources may be reused among nodes 2 to N before fusion. The peak memory usage increases after fusion. Therefore, the nodes cannot be fused.
Figure 4 Fusion condition 4 checked by CanFuse
Fusion Conditions Checked by Backend
- The loop axes of two AscGraphs can be mapped.
The first condition for fusion is whether the loop axes of two AscBackends can be mapped. If they are not the same set of loop axes, the fusion is meaningless. Lowering is separately performed for each AscBackend, and the loop axis IDs are independent. In this case, the axis IDs may be inconsistent. The following figure shows a typical scenario. After the reduction, AscBackend1 has one less loop axis. z1 in AscBackend2 is equivalent to z2 in AscBackend1, and z2 in AscBackend2 is equivalent to z3 in AscBackend1. To perform fusion, the loop axis in AscBackend2 must be updated to be the same as that in AscBackend1.
Figure 5 Fusion conditions checked by Backend
- Two AscGraphs must meet the group merge rules of Schedule.
Assume that axes are abstractly grouped into three groups: xgroup, ygroup, and rgroup.
- xgroup: an independent group introduced by the Concat operator. The axis before the Concat axis is xgroup, and the axis after the Concat axis is ygroup.
- ygroup: a loop axis of the Elementwise and Broadcast operators.
- rgroup: a set of reduced axes.
Each AscGraph belongs to a loop axis group (xgroup, ygroup, or rgroup). Based on the operator fusion rules, it can be determined whether two AscGraphs can be fused into a new group. Then, CanFuse determines whether the backend Schedule module supports fusion based on the rules. For details about group merge rules, see TilingCase Generation.