VF Loop Optimization

Hardware Loop Coding Specifications

The loop code must meet the hardware design requirements so that the compiler can identify the code and generate a hardware loop. The specific specifications are as follows:

• Iteration variable type

  The iteration variables of all loops in the VF must be of the uint16_t type.

• Start value and step

  The loop starts from 0.

  The iteration step must be 1.

• In loops, jump instructions are not allowed, such as jump instructions generated from conditional judgment statements like if/else and ternary operators.

  Using if/else statements in a VF loop hinders the generation of hardware loops. The compiler attempts to eliminate them, but this is best-effort and not guaranteed.

  The if control flow can be replaced by if constexpr or for(1). if constexpr is resolved at compile time and has no runtime overhead. However, the input parameters must be compile-time constants and cannot depend on runtime variables. for(1) can trigger compiler loop optimizations. Under the current hardware conditions, the performance of executing loops on the vector side is much better than that of conditional branch jumps.

  The following example illustrates a tail block processing scenario, where hasTail is 1 when the tail block size is not 0. for(1) is used to replace if(hasTail), improving the loop performance.

  1 2 3 4 5 6 7 8

  // [Negative example] The if statement is used. uint16_t tailK = srcK % floatRepSize; uint16_t hasTail = 0; // Use !!tailK to determine whether a tail block is generated. If the remainder of srcK % floatRepSize is 0, the bool value of hasTail is 0 (false). Otherwise, the bool value is 1 (true). hasTail = !!tailK; if(tailK > 0){ // Tail block processing content }

  1 2 3 4 5 6 7

  // [Positive example] Using for(1) instead of the if statement uint16_t tailK = srcK % floatRepSize; uint16_t hasTail = 0; hasTail = !!tailK; for(uint16_t i = 0 ; i < hasTail ; i++){ // Tail block processing content }

• Once the execution starts, the loop counter and boundary cannot be modified.
• To use the count of the outer loop as the loop boundary, move the outer loop counter to another register and then set it as the loop boundary.

The following examples illustrate how the compiler separately identifies and processes loops as hardware and software loops.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// [Positive example] A loop is optimized by the compiler into a hardware loop.
// Nested loop
for (uint16_t i = 0; i < LoopBound; i++) {            
    for (uint16_t j = 0; j < LoopBound; j++) {   
        for (uint16_t k = 0; k < LoopBound; k++) {   
            for (uint16_t m = 0; m < LoopBound; m++) {  
              // ...
            }
         }
     }
}

1
2
3
4
5
6
7
8
9

// [Negative example] A loop cannot be optimized by the compiler and forms a software loop.
for (uint16_t i = 0; i < LoopBound; i++) {     // Software loop. The loop contains an if statement.
    if(){
    }
}
 
for (uint16_t i = 2; i < LoopBound*3; i+=2){  // Software loop. The loop start value is not 0, and the loop step is not 1.
...
}

Optimization of Access to Member Variables in Loops

Within a VF, direct access to class member variables is not recommended. Direct access to class member variables is equivalent to moving content from the stack to a tensor register and accessing data on the tensor through the address. This will cause VF fusion to fail. In this case, you are advised to temporarily transfer parameters using local variables. The following is an example:

1
2
3
4
5
6
7
8

// [Negative example] Directly reading member variables
__aicore__ inline void SoftMaxGenericNDImpVF(__ubuf__ float* dstAddr, __ubuf__ float* sumAddr, __ubuf__ float* maxAddr,
    __ubuf__ float* srcAddr, __ubuf__ float* workAddr, const LastAxisShapeND originalSrcShape, const SoftMaxTiling tiling)
{
    for (uint16_t i = 0; i < (uint16_t)tiling.srcM; i++) {
        AscendC::ReduceMax(maxAddr + i * FLOAT_NUM_PER_BLK, srcAddr + i * tiling.srcK, workAddr, (uint16_t)originalSrcShape.k);
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// [Positive example] Passing member variables to be accessed in the VF through local variables
__aicore__ inline void SoftMaxGenericNDImpVF(__ubuf__ float* dstAddr, __ubuf__ float* sumAddr, __ubuf__ float* maxAddr,
    __ubuf__ float* srcAddr, __ubuf__ float* workAddr, const LastAxisShapeND originalSrcShape, const SoftMaxTiling tiling)
{
    uint16_t srcK = tiling.srcK;
    uint16_t srcM = tiling.srcM;
    uint16_t reduceK = FLOAT_NUM_PER_BLK;
    uint16_t originK = (uint16_t)originalSrcShape.k;
    for (uint16_t i = 0; i < (uint16_t)srcM; i++) {
        AscendC::ReduceMax(maxAddr + i * reduceK, srcAddr + i * srcK, workAddr, originK);
    }
}

Optimization of Instruction Distribution in Loops

Non-index-related statements in loops are reduced. Non-index-related statements in a for loop can be hoisted out of the loop to reduce the number of instructions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// [Negative example] The Duplicate statement is placed in the for loop and executed once in each iteration.
template<typename T>
__simd_vf__ inline void DuplicateVF(__ubuf__ T* dstAddr, T scalarValue, uint32_t oneRepeatSize, uint16_t repeatTimes)
{
    AscendC::Reg::RegTensor<T> dstReg;
    AscendC::Reg::MaskReg mask = AscendC::Reg::CreateMask<T>();  
    for (uint16_t i = 0; i < repeatTimes; i++) {
        AscendC::Reg::Duplicate(dstReg, scalarValue);
        AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg, mask);
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// [Positive example] The Duplicate statement is placed outside the for loop and is executed only once to reduce the number of instructions.
template<typename T>
__simd_vf__ inline void DuplicateVF(__ubuf__ T* dstAddr, T scalarValue, uint32_t oneRepeatSize, uint16_t repeatTimes)
{
    AscendC::Reg::RegTensor<T> dstReg;
    AscendC::Reg::MaskReg mask = AscendC::Reg::CreateMask<T>();  
    AscendC::Reg::Duplicate(dstReg, scalarValue);
    for (uint16_t i = 0; i < repeatTimes; i++) {
        AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg, mask);
    }
}

Optimization of Address Management for Loops

In a VF loop, when a movement instruction is used, the address offsets for move-in and move-out need to be computed, which introduces a large amount of scalar computation overhead. In the Atlas 350 Accelerator Card, address registers are introduced to effectively optimize the computation of address offsets. When the following address register generation mode is met, the compiler has the opportunity to generate address registers, thereby eliminating related scalar computation overhead and improving overall performance.

An address register supports up to four levels of loop addressing, as shown in the following figure.

The following is a code example that complies with the address register generation mode. The source operand address is accessed via up to four-dimensional addressing as shown in the figure above.

1
2
3
4
5
6
7
8
9

for(uint16_t i = 0;i < extent1; i++){
    for(uint16_t j = 0;j < extent2; j++){
        for(uint16_t k = 0;k < extent3; k++){
            for(uint16_t m = 0;m < extent4; m++){
               AscendC::Reg::LoadAlign(srcReg, srcAddr + i * const1 + j * const2 + k * const3 + m * const4);
            }
        }
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

AscendC::Reg::AddrReg aReg;
for(uint16_t i = 0;i < extent1; i++){
    for(uint16_t j = 0;j < extent2; j++){
        for(uint16_t k = 0;k < extent3; k++){
            for(uint16_t m = 0;m < extent4; m++){
                aReg = AscendC::Reg::CreateAddrReg(i, const1, j, const2, k, const3, m, const4);
                AscendC::Reg::LoadAlign(srcReg, srcAddr, aReg);
            }
        }
    }
}

The compiler performs pattern matching by identifying instructions such as LoadAlign and StoreAlign. When the code structure meets a specific optimization pattern, the compiler can potentially perform efficient optimization, thereby achieving optimal performance benefits. Conversely, if AddrReg is directly used to store offsets or other underlying interfaces are used, the compiler's global optimization capability may be limited.

In particular, when four-level loops form continuous access, address management can be simplified to a one-dimensional mode, achieving more efficient movement optimization and further improving the locality and execution efficiency of data access.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Enabling vector address generation instruction optimization
__simd_vf__ inline void ComputeModeVF(__ubuf__ T* dstAddr, __ubuf__ T* srcAddr, uint32_t oneRepeatSize, uint16_t repeatTimes)
{
    AscendC::Reg::RegTensor<T> dstReg;
    AscendC::Reg::MaskReg mask = AscendC::Reg::CreateMask<T>();
    for (uint16_t i = 0; i < repeatTimes; ++i) {
        AscendC::Reg::LoadAlign(dstReg, srcAddr+ i * oneRepeatSize);
        AscendC::Reg::StoreAlign(dstAddr, dstReg, i * oneRepeatSize, mask);
    }
}