Overview

Function Description

SuperKernel is an operator fusion technology that operates at the binary level, distinguishing itself from traditional source code-level fusion. Its core principle lies in performing comprehensive optimization on already-compiled binary kernel functions (kernels). Through intelligent scheduling and composition, it combines multiple independent kernel functions (sub-kernels) into an efficient "SuperKernel" – invoking multiple other kernel functions collectively as sub-functions. Compared to individual operator dispatch, SuperKernel significantly reduces task scheduling wait time and scheduling overhead, while effectively utilizing idle resources between sub-tasks to further optimize overall operator execution efficiency and reduce overhead.

To improve the flexibility of SuperKernel fusion, a SuperKernel marking mechanism based on scope tags is introduced. You can manually specify the operators to be fused into the same SuperKernel, and manually mark the operators that do not participate in the fusion. The no fusion tag has the highest priority.

Currently, this feature is supported only by Atlas A3 training products/Atlas A3 inference products.

Principles

When detecting that a user manually marks a scope, the system performs SuperKernel fusion based on that scope. In terms of the SuperKernel generation policy, the system groups operators based on their tag sets. Operators with the same tag set are grouped together and proceed to the subsequent SuperKernel fusion process. The workflow is as follows:

• Record marking information.

  The system traverses all operators and records the tag set (operators that belong to the same marked scope) assigned to each operator by the user.

• Group by tag set.

  All operators are grouped based on the principle of having the same tag set. Only operators with the same tag set can be grouped into the same group.

• Trigger the fusion process.

  Operators in each group proceed through the SuperKernel fusion logic (such as sk1 and sk2 in Figure 1).

The following two figures show examples of marking the SuperKernel fusion scope.

Restrictions

If there are operators that cannot be fused within the marked scope, the system generates the first SuperKernel process, automatically skips the operators, and proceeds to the second SuperKernel fusion process. The marked scope does not represent the final fusion result.