Documentation
Download

Instruction Dual-Issue

Instruction dual-issue allows a processor to simultaneously issue two instructions to the execution unit for processing within a clock cycle. To implement this capability, the following conditions must be met:

  • There is no data dependency between the two instructions. (The dependency means that the latter instruction to be executed requires the result generated by the former instruction.)
  • The hardware has sufficient execution resources.

This mechanism, one of the important foundations for implementing instruction-level parallelism, improves the instruction processing efficiency of the processor per unit time without changing the program logic.

In the following example, instructions in VLoop-1 are executed 16 times. Because the four instructions in each loop have data dependencies, the depth of the execution queue is 64. The following figure shows the concurrent execution sequence of 64 instructions. LoadAlign_0 and LoadAlign_1 have no dependency and can be executed concurrently. The four instructions in the black-framed box are eligible for concurrent execution. In practice, two of them are selected out of order for execution.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
 
for (uint16_t i = 0; i < 16; ++i) { // VLoop-1
    // Executed 16 times, with four instructions each time
    //Data dependency: Adds depends on LoadAlign, Mul depends on Adds, and so on.
    mask = AscendC::Reg::UpdateMask<T>(count);
    int16_t scalar = 2;
    AscendC::Reg::LoadAlign(srcReg, src0Addr + i * oneRepeatSize);
    AscendC::Reg::Adds(dstReg1, srcReg, scalar , mask);
    AscendC::Reg::Mul(dstReg2, dstReg1, srcReg, mask);
    AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg2, mask);
}
Figure 1 Execution queue and instruction execution sequence

When implementing an operator, developers usually structure the code flow in the order of "data loading→ computation → result storage". This approach works well when register resources are sufficient, but will lead to severe problems once resources are insufficient. When dependencies exist between multiple compute instructions, these stalls accumulate in the execution queue, preventing subsequent instructions from being issued in time.

When programming, developers should ensure that there are sufficient independent concurrent instructions in the queue to efficiently utilize the dual-issue feature of hardware. Performance can be improved by properly splitting VF loops and manually controlling loop unrolling.

Properly Splitting VF Loops

Writing an excessively long VF that packs all computations into one for loop is not the optimal approach. Instead, intermediate results should be properly moved to Unified Buffer to reduce data dependencies.

Before optimization: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
 
for (uint16_t i = 0; i < 32; ++i) { // VLoop-1
    // Data dependency: The input of each Adds instruction depends on the result of the previous instruction.
    mask = AscendC::Reg::UpdateMask<T>(count);
    AscendC::Reg::LoadAlign(srcReg0, src0Addr + i * oneRepeatSize);
    AscendC::Reg::LoadAlign(srcReg1, src1Addr + i * oneRepeatSize);
    AscendC::Reg::Add(dstReg, srcReg0, srcReg1, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg, mask);
}

After optimization:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
 
for (uint16_t i = 0; i < 32; ++i) { // VLoop-1
    mask = AscendC::Reg::UpdateMask<T>(count);
    AscendC::Reg::LoadAlign(srcReg0, src0Addr + i * oneRepeatSize);
    AscendC::Reg::LoadAlign(srcReg1, src1Addr + i * oneRepeatSize);
    AscendC::Reg::Add(dstReg, srcReg0, srcReg1, mask);
    AscendC::Reg::Adds(dstReg, dstReg1, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg1, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg1, 10, mask);
    AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg, mask);
}
for (uint16_t i = 0; i < 32; ++i) { // VLoop-2
    mask = AscendC::Reg::UpdateMask<T>(count);
    AscendC::Reg::LoadAlign(dstReg, dstAddr + i * oneRepeatSize);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::Adds(dstReg, dstReg, 10, mask);
    AscendC::Reg::StoreAlign(dstAddr + i * oneRepeatSize, dstReg, mask);
}

Manual Control of Loop Splitting

If there are too many instructions with dependencies in a loop, all instructions of for(i = n) and for(i = n+1) cannot be loaded in the queue at the same time. In this case, even if there is no dependency between loops, the dual-issue feature cannot be enabled, and instructions cannot be executed concurrently. Manual loop unrolling can be applied, which helps align with hardware out-of-order execution characteristics, creating more opportunities for instruction execution. In addition, it reduces instruction stalls due to unavailable register resources.

1
2
3
4
5
6
7
8
 
for (uint16_t i = 0; i < 32; ++i) { 
//There is no dependency between the for loops. i=0 and i=1 can be executed concurrently. However, due to excessive data dependency instructions within the loop, the instructions for i=1 cannot be loaded into the execution queue.
    AscendC::Reg::LoadAlign(srcReg, srcAddr, offset);
    AscendC::Reg::Adds(dstReg0, srcReg, 10, mask);
    AscendC::Reg::Muls(dstReg1 dstReg0, 20, mask);
    ... // More than 64 instructions with data dependencies
    AscendC::Reg::StoreAlign(dstAddr, dstReg1, offset, mask);
}

After unrolling

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
 
for (uint16_t i = 0; i < 8; ++i) { // Unrolling a loop of 32 iterations by a factor of 4
    AscendC::Reg::LoadAlign(srcReg0, srcAddr, offset);
    AscendC::Reg::LoadAlign(srcReg1, srcAddr, offset);
    AscendC::Reg::LoadAlign(srcReg2, srcAddr, offset);
    AscendC::Reg::LoadAlign(srcReg3, srcAddr, offset);
    AscendC::Reg::Adds(...)
    AscendC::Reg::Adds(...)
    AscendC::Reg::Adds(...)
    AscendC::Reg::Adds(...)
    AscendC::Reg::Muls(...)
    AscendC::Reg::Muls(...)
    AscendC::Reg::Muls(...)
    AscendC::Reg::Muls(...)
    ...
    AscendC::Reg::StoreAlign(...)
    AscendC::Reg::StoreAlign(...)
    AscendC::Reg::StoreAlign(...)
    AscendC::Reg::StoreAlign(...)
}

Avoiding the Increase in the Number of Dependent Instructions in the Execution Queue Due to Excessive Number of Registers

In the same VF, the hardware can process a maximum of 32 RegTensor registers at the same time. If the upper limit is exceeded, the compiler introduces data spilling and reloading, as well as synchronization instructions, severely reducing the operator execution efficiency. Similarly, in the same VF, the number of MaskReg registers cannot exceed 8, and the number of UnalignRegForLoad and UnalignRegForStore registers cannot exceed 4, respectively. Otherwise, register overflow may occur, leading to performance deterioration.

Optimization solutions:

  • Use Boolean algebra operations. For example, !(a && b) can be simplified to !a || !b, and !(a || b) can be simplified to !a && !b.
  • Perform instruction reordering based on equivalence transformations to save registers.

The examples are used to determine whether two double-type data values are equal. Two typical scenarios are involved: one data value is NaN, and the two data values are separately +0 and -0.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
template<typename T = Reg::DefaultType, CMPMODE mode = CMPMODE::EQ, typename RegT>
__simd_caller__ inline void CompareDoubleImpl(Reg::MaskReg &dstMask, RegT &srcReg0, RegT &srcReg1, Reg::MaskReg &mask)
{
    using ActualT = typename RegT::ActualT;
    static_assert(SupportType<ActualT, double, uint64_t>(), "CompareDoubleImpl only support double and uint64_t type");

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> tmpSrcReg0 = (Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo>&)srcReg0;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> tmpSrcReg1 = (Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo>&)srcReg1;

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> exponent0;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> exponent1;

    Reg::ShiftRights(exponent0, tmpSrcReg0, static_cast<int16_t>(52), mask);
    Reg::ShiftRights(exponent1, tmpSrcReg1, static_cast<int16_t>(52), mask);

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarExponent;
    Reg::Duplicate(scalarExponent, static_cast<uint64_t>(0x7ff), mask);
    Reg::And(exponent0, exponent0, scalarExponent, mask);
    Reg::And(exponent1, exponent1, scalarExponent, mask);

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> mantissa0, mantissa1;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarMantissa;
    Reg::Duplicate(scalarMantissa, static_cast<uint64_t>(0xfffffffffffff), mask);
    Reg::And(mantissa0, tmpSrcReg0, scalarMantissa, mask);
    Reg::And(mantissa1, tmpSrcReg1, scalarMantissa, mask);

    Reg::MaskReg tmpMask0, tmpMask1;
    Reg::Compares(tmpMask0, exponent0, 0x7ff, mask);
    Reg::Compares(dstMask, exponent1, 0x7ff, mask);
    Reg::MaskAnd(dstMask, tmpMask0, dstMask, mask);
    // dstMask indicates that the mantissa bits of two double-type data values are not all 0s. Check whether the mantissa bits of the two numbers are not 0s. If they are all 0s, dstMask bits are 0s. Then, perform the OR operation. If tmpMask bits are 1s, the two numbers are not all 0.
    Reg::MaskReg tmpMask1;
    Reg::Compares<uint64_t, CMPMODE::NE>(tmpMask1, mantissa0, 0, mask);
    Reg::Compares<uint64_t, CMPMODE::NE>(tmpMask0, mantissa1, 0, mask);
    Reg::MaskOr(tmpMask0, tmpMask1, tmpMask0, mask);
    // [Negative example] In the NaN scenario where exponent bits are all 1s and mantissa bits are not all 0s, the !(a&&b) formula is used. An additional MaskReg, noNaNMask, needs to be requested to record the intermediate result of a&&b.
    Reg::MaskReg noNaNMask;
    Reg::MaskAnd(noNaNMask, dstMask, tmpMask0, mask);
    Reg::MaskNot(noNaNMask, noNaNMask, mask);
    // In the typical scenario where the two data values are +0 and -0
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> unsignedPart0, unsignedPart1;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarUnsignedPart;
    Reg::Duplicate(scalarUnsignedPart, static_cast<uint64_t>(0x7fffffffffffff), mask);
    Reg::And(unsignedPart0, tmpSrcReg0, scalarUnsignedPart, mask);
    Reg::And(unsignedPart1, tmpSrcReg1, scalarUnsignedPart, mask);
    // [Negative example] Check whether the two unsigned numbers are 0, and then perform the AND operation on the results.
    Reg::Compares<uint64_t, CMPMODE::EQ>(tmpMask0, unsignedPart0, 0, mask);
    Reg::Compares<uint64_t, CMPMODE::EQ>(dstMask, unsignedPart1, 0, mask);
    Reg::MaskAnd(tmpMask0, tmpMask0, dstMask, mask);

    Reg::Compare(dstMask, tmpSrcReg0, tmpSrcReg1, mask);
    Reg::MaskAnd(dstMask, dstMask, noNaNMask, mask);
    Reg::MaskOr(dstMask, dstMask, tmpMask0, mask);
}
After optimization: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
 
template <typename T = Reg::DefaultType, CMPMODE mode = CMPMODE::EQ, typename RegT>
_simd_caller inline void CompareDoubleImpl(Reg::MaskReg &dstMask, Reg &srcReg0, Reg &srcReg1, Reg::MaskReg &mask)
{
    using ActualT = typename RegT::ActualT;
    static_assert(SupportType<ActualT, double, uint64_t>(), "CompareDoubleImpl only support double and uint64_t type");

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> tmpSrcReg0 = (Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo>&)srcReg0;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> tmpSrcReg1 = (Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo>&)srcReg1;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> exponent0;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> exponent1;

    Reg::ShiftRights(exponent0, tmpSrcReg0, static_cast<int16_t>(52), mask);
    Reg::ShiftRights(exponent1, tmpSrcReg1, static_cast<int16_t>(52), mask);
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarExponent;
    Reg::Duplicate(scalarExponent, static_cast<uint64_t>(0x7ff), mask);
    Reg::And(exponent0, exponent0, scalarExponent, mask);
    Reg::And(exponent1, exponent1, scalarExponent, mask);

    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> mantissa0, mantissa1;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarMantissa;
    Reg::Duplicate(scalarMantissa, static_cast<uint64_t>(0xfffffffffffff), mask);
    Reg::And(mantissa0, tmpSrcReg0, scalarMantissa, mask);
    Reg::And(mantissa1, tmpSrcReg1, scalarMantissa, mask);

    Reg::MaskReg tmpMask0, tmpMask1;
    Reg::Compares(tmpMask0, exponent0, 0x7ff, mask);
    Reg::Compares(dstMask, exponent1, 0x7ff, tmpMask0);
    Reg::MaskNot(dstMask, dstMask, mask);
    Reg::Compares<uint64_t, CMPMODE::EQ>(tmpMask1, mantissa0, 0, mask);
    Reg::Compares<uint64_t, CMPMODE::EQ>(tmpMask0, mantissa1, 0, tmpMask1);
    // [Positive example] Simplify !(a&&b) to !a||!b, indicating that the evaluation can be performed properly without requesting additional registers when the exponent bits are not all 1s or the tmpMask0 bits are all 0s.
    Reg::MaskOr(tmpMask0, tmpMask0, dstMask, mask);

    Reg::Compare(dstMask, tmpSrcReg0, tmpSrcReg1, mask);
    Reg::MaskAnd(dstMask, dstMask, tmpMask0, mask);
    // In the typical scenario where the two data values are +0 and -0
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> unsignedPart0, unsignedPart1;
    Reg::RegTensor<uint64_t, Reg::RegTraitNumTwo> scalarUnsignedPart;
    Reg::Duplicate(scalarUnsignedPart, static_cast<uint64_t>(0x7fffffffffffff), mask);
    Reg::And(unsignedPart0, tmpSrcReg0, scalarUnsignedPart, mask);
    Reg::And(unsignedPart1, tmpSrcReg1, scalarUnsignedPart, mask);
    // [Positive example] The mask generated from the evaluation of unsignedPart1 is replaced with tmpMask0, which is equivalent to performing the AND operation after unsignedPart0 and unsignedPart1 are evaluated. Compared with the negative example, one MaskAnd instruction is omitted.
    Reg::Compares<uint64_t, CMPMODE::EQ>(tmpMask0, unsignedPart0, 0, mask);
    Reg::Compares<uint64_t, CMPMODE::EQ>(tmpMask1, unsignedPart1, 0, tmpMask0);
    Reg::MaskOr(dstMask, dstMask, tmpMask0, mask);
}
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details