VF Fusion

[Priority] High

[Description] VF fusion is to fuse multiple VF functions in the code into one VF function, effectively improving performance. The VF fusion feature is an optimization feature. By leveraging the Loop Fuse algorithm, automatic VF fusion converts VFs into a loop form, fuses control-flow-equivalent VFs, and then restores them to the VF form. The compiler first performs a validity check before fusion to determine whether the two VFs are equivalent, whether the intermediate code on the main side can be executed within the VFs, and whether positive benefits can be generated after fusion (without causing register spilling due to parameter passing or excessively large VF code size). If the VF fusion conditions are met, the compiler automatically performs VF fusion for optimization. To ensure that the execution logic and semantics of the VF obtained after fusion are consistent with those before fusion, the compiler conservatively inserts synchronization instructions between the original two VFs. The compiler also attempts to hoist and fuse instructions in the VF obtained after fusion to optimize the VF code. The fusion policy is to fuse whenever possible. Users can code in a way that satisfies fusion legality checks to increase VF fusion opportunities.

VF Fusion Principles

VF fusion can be divided into three phases: VF shallow fusion, VF deep fusion, and intra-VF automatic synchronization.

VF shallow fusion: The compiler first analyzes whether the control flows of the two VFs are equivalent and constructs a cost model to analyze whether there are positive benefits. If the VF fusion conditions are met, the compiler fuses the control flows outside the VFs into the VFs, hardens the software loops outside the VFs into hardware loops inside the VFs, and then enables the basic capability of automatic VF fusion to fuse the two VFs into one VF. This lays a foundation for subsequent VF deep fusion.

VF deep fusion: VF deep fusion continues to fuse hardware loops within VFs. This reduces the startup overhead of hardware loops, significantly reduces redundant load/store operations, and maximizes register reuse.

Intra-VF automatic synchronization: The compiler accurately inserts necessary synchronization instructions and deletes redundant synchronization instructions, greatly unleashing the hardware out-of-order (OOO) capability. Users do not need to manually insert synchronization instructions, significantly simplifying coding.

VF Fusion Compilation Guide

Automatic fusion of multiple VF functions: If the control flows of multiple VF functions are equivalent and the loops in the VF functions are all hardware loops, the compiler performs VF fusion for optimization.

[Positive example] The VF functions DivVF and AddVF are fused into one VF function by the compiler, and redundant load/store instructions are optimized.

        
         
           
           
             template<typename T>
__simd_vf__ inline void DivVF(__ubuf__ T* dstAddr, __ubuf__ T* srcAddr, uint32_t count, uint32_t repeatTime, uint32_t oneRepNum){
    AscendC::Reg::MaskReg mask;
    AscendC::Reg::RegTensor<T> reg0, reg1, reg2;
    constexpr float num = 1.0f;
    for(uint16_t j = 0; j < repeatTime; ++j){
        mask = AscendC::Reg::UpdateMask<T>(count);
        AscendC::Reg::LoadAlign(reg0, srcAddr + j * oneRepNum);
        AscendC::Reg::Duplicate(reg1, num, mask);
        AscendC::Reg::Div(reg2, reg1, reg0, mask);
        AscendC::Reg::StoreAlign(dstAddr + j * oneRepNum, reg2, mask);
    }
}
template<typename T>
__simd_vf__ inline void AddVF(__ubuf__ T* dstAddr, __ubuf__ T* srcAddr, uint32_t count, uint32_t repeatTime, uint32_t oneRepNum){
    AscendC::Reg::MaskReg mask;
    AscendC::Reg::RegTensor<T> srcReg;
    AscendC::Reg::RegTensor<T> dstReg;
    constexpr float num = 1.0f;
    for(uint16_t j = 0; j < repeatTime; ++j){
        mask = AscendC::Reg::UpdateMask<T>(count);
        AscendC::Reg::LoadAlign(srcReg, srcAddr + j * oneRepNum);
        AscendC::Reg::Adds(dstReg, srcReg, num, mask);
        AscendC::Reg::StoreAlign(dstAddr + j * oneRepNum, dstReg, mask);
    }
}
template<typename T>
class Kernel {
    public:
    __aicore__ inline Kernel() = default;
    __aicore__ inline void Init(GM_ADDR x, GM_ADDR y, uint32_t count, AscendC::TPipe* pipeIn){
        // ... 
 
    }
    __aicore__ inline void CopyIn(){
        // ... 
    }
    __aicore__ inline void Compute(){
        AscendC::LocalTensor<T> xLocal = inQueueX.DeQue<T>();
        AscendC::LocalTensor<T> yLocal = outQueueY.AllocTensor<T>();
        AscendC::DataCopy(yLocal, xLocal, count);
        __ubuf__ T* srcAddr = reinterpret_cast<__ubuf__ T*>(xLocal.GetPhyAddr());
        __ubuf__ T* dstAddr = reinterpret_cast<__ubuf__ T*>(yLocal.GetPhyAddr());
        constexpr uint32_t oneRepNum = 256 / sizeof(T);
        uint32_t repeatTime =  count / oneRepNum;
        DivVF(dstAddr, srcAddr, count, repeatTime, oneRepNum);
        AddVF(dstAddr, dstAddr, count, repeatTime, oneRepNum);
        outQueueY.EnQue<T>(yLocal);
    }
    __aicore__ inline void CopyOut(){
        // ... 
    }
    __aicore__ inline void Process(){
        CopyIn();
        Compute();
        CopyOut();
    }
    private:
    AscendC::TPipe* pipe = nullptr;
    uint32_t count;
    AscendC::GlobalTensor<T> xGm;
    AscendC::GlobalTensor<T> yGm;
    AscendC::TQue<AscendC::TPosition::VECIN, 1> inQueueX;
    AscendC::TQue<AscendC::TPosition::VECOUT, 1> outQueueY;
};

            

          

        
       

Using contiguous compute APIs: The basic APIs abstract hardware capabilities, open up chip capabilities, and ensure completeness and compatibility. Basic APIs can be classified into two types based on the data operation methods:

Contiguous compute APIs: These APIs support computation of the first n data elements of a tensor. They compute n contiguous data elements of the source operand and contiguously write the data to the destination operand to solve the contiguous compute problem of the one-dimensional tensor.
High-dimensional sharding APIs: These APIs support repeat and stride. They are flexible compute APIs that provide programming capabilities fully equivalent to those of built-in APIs, fully leveraging hardware advantages. They support operations on parameters such as DataBlock Stride, Repeat Stride, and Mask.

In VF fusion for optimization, it is recommended that operators be written using contiguous compute APIs to fully utilize the VF fusion capability. Compared with high-dimensional sharding APIs, contiguous compute APIs enable the compiler to better analyze VF fusion, making it easier to meet the VF fusion conditions. Contiguous compute APIs can be used to write operators with better performance.

[Negative example] When high-dimensional sharding APIs are used to write operators, the compiler cannot perform VF fusion on the Add and Mul functions due to the impact of complex computing logic during VF fusion analysis.

        
             template<typename T>
class Kernel {
    public:
    // ...
    __aicore__ inline void Compute(){
        AscendC::LocalTensor<T> xLocal = inQueueX.DeQue<T>();
        AscendC::LocalTensor<T> yLocal = outQueueY.AllocTensor<T>();
        AscendC::DataCopy(yLocal, xLocal, inner * outter);
        uint64_t mask = 128;
        AscendC::Add(yLocal, xLocal, xLocal, mask, 4, { 1, 1, 1, 8, 8, 8 });
        AscendC::Mul(yLocal, yLocal, xLocal, mask, 4, { 1, 1, 1, 8, 8, 8 });
        outQueueY.EnQue<T>(yLocal);
    }
    // ...
};

[Positive example] The contiguous compute APIs are used. After analysis, the compiler determines that the Add and Mul functions meet the VF fusion requirements and fuses them into a single VF function.

         
              template<typename T>
class Kernel {
    public:
    // ...
    __aicore__ inline void Compute(){
        AscendC::LocalTensor<T> xLocal = inQueueX.DeQue<T>();
        AscendC::LocalTensor<T> yLocal = outQueueY.AllocTensor<T>();
        AscendC::DataCopy(yLocal, xLocal, inner * outter);
        AscendC::Add(yLocal, xLocal, xLocal, count);
        AscendC::Mul(yLocal, yLocal, xLocal, count);
        outQueueY.EnQue<T>(yLocal);
    }
    // ...
};

Parent topic: VF Performance Optimization