SuperKernel Development

Supporting SuperKernel for Custom Operators

The development process for custom operators to support SuperKernel is similar to that of common operators. However, you need to pay attention to some specific restrictions (as described below). Currently, the SuperKernel feature can be used only in the PyTorch framework. Therefore, after integrating operators into a GE graph, you need to integrate them into a PyTorch graph by referring to Integrating Custom Operators to a Graph in PyTorch Graph Mode User Guide (TorchAir). In addition, TorchAir provides the capability of calibrating the scope of SuperKernel. You can annotate operators and configure optimizations within the fusion scope based on actual service requirements. For details, see "GE Graph Mode > GE Graph Mode Features > Marking the SuperKernel Scope in a Graph" in PyTorch Graph Mode User Guide (TorchAir).

The specific restrictions during development are as follows:

• If full-core synchronization is implemented for a custom operator, ensure that the number of cores launched by the sub kernel (that is, the operator) is the same as that launched by the SuperKernel. If the sub kernel launches fewer cores than the SuperKernel, full-core synchronization will wait for all cores to complete, causing a stall and timeout.

  Note: The number of cores launched by the SuperKernel is the maximum number of cores launched by the sub kernel. Assume that the SuperKernel includes operator a (launching four cores) and operator b (launching two cores). In this case, the number of cores launched by the SuperKernel is four.

  • When SyncAll is used, you can enable feed-sync-all during the calibration of the SuperKernel scope to address this problem. In this case, the system inserts the SyncAll instruction into the remaining cores of the sub kernel in the SuperKernel to prevent timeout.
  • If hardware synchronization APIs CrossCoreSetFlag and CrossCoreWaitFlag are used to implement full-core synchronization, the number of cores launched by the sub kernel must be the same as that launched by the SuperKernel.
• If the kernel type of a custom operator is set to KERNEL_TYPE_MIX_AIC_1_1, note that the SuperKernel adjusts its launch ratio based on information such as the number of launched cores. In this case, the operator can also support the 1:2 launch ratio in the SuperKernel to ensure that the hardware synchronization between the AIC and AIV is correctly performed. For example, if the operator uses the hardware synchronization APIs (CrossCoreSetFlag and CrossCoreWaitFlag) between the AIC and AIV, do not specify some AIV cores to call the hardware synchronization APIs separately. Instead, ensure that all AIV cores call the hardware synchronization APIs to prevent suspension and timeout caused by mismatched number of hardware synchronization operations. When using a Matmul high-level API, ensure that only one AIV0 core calls the Matmul API in the operator logic. This prevents suspension and timeout caused by the failure to receive AIV1 core messages after two AIV cores are started.

• When developing a custom operator, you must ensure that the DataCacheCleanAndInvalid instruction is correctly inserted for all GM read and write operations performed by the Scalar Unit as required. In the single-operator compilation scenario, the BiSheng Compiler automatically appends the DataCacheCleanAndInvalid instruction at the end of the operator to update the entire DCache. However, in a SuperKernel, sub kernels are processed as common functions, and the compiler does not automatically insert this instruction to ensure data cache consistency. You need to ensure that errors are not caused by the change of the fault tolerance mechanism.

  For performance considerations, the cache update mechanism in the SuperKernel scenario is as follows:

  If you call the GetValue and SetValue APIs of GlobalTensor to read and write scalars in the GM, the DataCacheCleanAndInvalid instruction is automatically inserted into the two APIs during SuperKernel compilation to update a single cache line, ensuring data cache consistency. The DataCacheCleanAndInvalid instruction is not inserted before and after the sub kernel is called.

  However, note that excessive calls to GetValue and SetValue will cause performance deterioration in the SuperKernel scenario. Therefore, you should minimize the calls to these APIs. For operators with excessive deterioration, SuperKernel provides compilation options dcci-before-kernel-start, dcci-after-kernel-start, and dcci-disable-on-kernel. You can disable the cache update instruction automatically inserted into GetValue/SetValue of a specified operator to improve model performance. The model user determines whether to insert the instruction for updating the entire DCache before or after the SuperKernel calls the operator. For details about the compilation options, see the description of the compilation options in SuperKernel range calibration in the graph.

  In particular, in the tiling offload scenario, binary reuse optimization is usually involved, and the preceding cache update mechanism cannot be selected online. The SuperKernel framework inserts the DataCacheCleanAndInvalid instruction before and after each sub kernel is called to update the entire DCache. Cache is not automatically updated in GetValue and SetValue.

  The following figure shows the cache update mechanism.

  

• When the GetBlockNum API is called in a sub kernel to obtain the number of cores, the returned number of cores remains unchanged regardless of whether the sub kernel is fused into a SuperKernel or how many cores the SuperKernel launches. Therefore, you can use this API in the same way as you would when developing a common operator, without needing to pay special attention to the number of cores launched by the SuperKernel.
• In the Atlas A3 training product / Atlas A3 inference product , if SuperKernel is disabled, the AscendC::PipeBarrier<PIPE_ALL>() instruction is inserted at the end of the TPipe::Destroy API to ensure pipeline synchronization between multiple TPipes. Most operators in the model use only one TPipe object, and the Destroy function is called during object destruction. To prevent SetNextTaskStart and WaitPreTaskEnd performance from being affected, the AscendC::PipeBarrier<PIPE_ALL>() instruction inserted in TPipe::Destroy is disabled by default in the SuperKernel scenario. Therefore, when an operator requires multiple TPipe objects and the Destroy function is manually called, you need to ensure pipeline synchronization between TPipe objects by yourself.

Performance Optimization Suggestions

• Inter-task synchronization

  In addition, during programming on the kernel, you can call SetNextTaskStart and WaitPreTaskEnd to further improve performance.

  • After SetNextTaskStart is called, the instructions that follow can be executed in parallel with other subsequent sub kernels, improving the overall performance. As shown in Figure 1, the SuperKernel calls sub kernels in sequence. To prevent data interference between sub kernels, inter-operator synchronization is inserted between sub kernels for order-preserving. After sub kernel_N-1 calls this API, subsequent instructions are executed in parallel with sub kernel_N.
    Figure 1 Parallelism implemented through SetNextTaskStart
    
  • After WaitPreTaskEnd is called, the preceding instructions can be executed in parallel with earlier sub kernels, improving the overall performance. As shown in Figure 2, the SuperKernel calls sub kernels in sequence. To prevent data interference between sub kernels, inter-operator synchronization is inserted between sub kernels for order-preserving. After sub kernel_N+1 calls this API, its preceding instructions are executed in parallel with sub kernel_N.
    Figure 2 Parallelism implemented through WaitPreTaskEnd
    

• In the tiling offload scenario, you can use the --op_relocatable_kernel_binary compilation option to enable binary reuse optimization, improving compilation performance. For details, see LINK.