Optimization in Contiguous Unaligned Scenarios

Optimization Suggestions for Contiguous Unaligned Data Movement

• Contiguous aligned data movement is preferred to avoid extra overhead and improve overall performance.
• Due to hardware support limitations, you are advised not to use AddrReg to store offsets during contiguous unaligned data movement. It is recommended that the uint32_t type be used to store offsets and the post update mode be used for data movement. In this scenario, the address of the source operand in Unified Buffer is automatically updated each time the interface is called. Subsequent optimization suggestions are provided based on this scenario.
• During contiguous unaligned data move-in, the unaligned register caches subsequent unaligned data each time after data is moved from Unified Buffer to RegTensor. During the next movement, the data in the unaligned register is written to RegTensor. Therefore, the initialization for contiguous unaligned data move-in needs to be performed only once and should be moved out of the for loop.
• During contiguous unaligned data move-out, the unaligned register caches subsequent unaligned data each time after data is moved from RegTensor to Unified Buffer. During the next movement, the data in the unaligned register is written to Unified Buffer. Therefore, post-processing for contiguous unaligned data move-out needs to be performed only once and should be moved out of the for loop.

[Positive Example]

1
2
3
4
5
6
7
8

// Initialization for unaligned data move-in is moved out of the for loop. uint32_t is used to manage the offset.
AscendC::Reg::LoadUnAlignPre(ureg0, srcAddr);
for (uint16_t i = 0; i < repeatTimes; ++i) {
     AscendC::Reg::LoadUnAlign(srcReg, ureg0, srcAddr + i * postUpdateStride);
     AscendC::Reg::StoreUnAlign(dstAddr, srcReg, ureg1, postUpdateStride);
}
// Post-processing for unaligned data move-out is moved out of the for loop.
AscendC::Reg::StoreUnAlignPost(dstAddr, ureg1, 0);

[Negative Example]

1
2
3
4
5
6

for (uint16_t i = 0; i < repeatTimes; ++i) {
     AscendC::Reg::LoadUnAlignPre(ureg0, srcAddr + i * postUpdateStride);
     AscendC::Reg::LoadUnAlign(srcReg, ureg0, srcAddr + i * postUpdateStride);
     AscendC::Reg::StoreUnAlign(dstAddr, srcReg, ureg1, postUpdateStride);
     AscendC::Reg::StoreUnAlignPost(dstAddr, ureg1, 0);
}

Introduction to Contiguous Unaligned Data Movement Interfaces

To improve the capability of processing irregular memory addresses, Reg vector computation supports access to non-32-byte-aligned addresses during data movement. This is applicable to scenarios with unaligned data movement from Unified Buffer to RegTensor or from RegTensor to Unified Buffer. To reduce the performance overhead caused by unaligned access, RegBase introduces an unaligned register-based cache mechanism. This mechanism uses the unaligned registers UnalignRegForLoad and UnalignRegForStore as temporary buffers to temporarily store data across alignment boundaries, thereby achieving efficient contiguous unaligned data transmission.

Before reading an unaligned address, perform initialization through LoadUnAlignPre before using LoadUnAlign. When writing an unaligned address, use StoreUnAlign first and then StoreUnAlignPost for post-processing. The move-in and move-out interfaces with offsets being managed using uint32_t and the maskReg data move-out interfaces are as follows:

• Contiguous unaligned data move-in, in which uint32_t is used to store the amount of moved data

  1 2	__simd_callee__ inline void LoadUnAlignPre(UnalignRegForLoad& ureg, __ubuf__ T* srcAddr); __simd_callee__ inline void LoadUnAlign(U& dstReg, UnalignRegForLoad& ureg, __ubuf__ T*& srcAddr, uint32_t postUpdateStride);

• Contiguous unaligned data move-out, in which uint32_t is used to store the offset

  1 2	__simd_callee__ inline void StoreUnAlign(__ubuf__ T*& dstAddr, U& srcReg, UnalignRegForStore& ureg, uint32_t postUpdateStride); __simd_callee__ inline void StoreUnAlignPost(__ubuf__ T*& dstAddr, UnalignRegForStore& ureg, int32_t postUpdateStride);

• Contiguous unaligned data move-out from maskReg to Unified Buffer

  1 2	__simd_callee__ inline void StoreUnAlign(__ubuf__ T*& dstAddr, MaskReg& mask, UnalignRegForStore& ureg); __simd_callee__ inline void StoreUnAlignPost(__ubuf__ T*& dstAddr, UnalignRegForStore& ureg, int32_t postUpdateStride);

Principles of Unaligned Registers

This part describes how unaligned registers work during unaligned data movement and explains the internal principles in detail to help developers understand why the preceding performance optimization suggestions can bring performance benefits. This part first introduces how contiguous unaligned data move-in, contiguous unaligned data move-out, and contiguous unaligned data move-out for MaskReg are implemented, and then uses an example to describe these key processes work together.

• Unaligned data move-in

  As shown in the following figure, data is read from Unified Buffer addresses srcAddr to 304 and moved to the destination register dstReg (256 bytes). The process is as follows:

  • LoadUnAlignPre is called to initialize unaligned data move-in. The unaligned register ureg caches the valid data of Unified Buffer addresses 32 to 64 as the front-end data cache for subsequent unaligned access.
  • LoadUnAlign is called. Hardware instructions are executed to move the data of Unified Buffer addresses 64 to 320 to the temporary register tmpReg, concatenate the data corresponding to addresses srcAddr to 64 in ureg with the data corresponding to addresses 64 to 304 in tmpReg, and write the result to dstReg. After the movement, the data of Unified Buffer addresses 288 to 320 is written to ureg. During contiguous unaligned data move-in, LoadUnAlign caches subsequent unaligned data to ureg. Therefore, LoadUnAlignPre does not need to be called again for the next move-in, and it only needs to be called once before the first move-in, optimizing the performance of unaligned data move-in.
    Figure 1 Unaligned data move-in

    

• Unaligned data move-out

  The unaligned data in the source register srcReg is written to the Unified Buffer address dstAddr. Based on the current status of ureg, two scenarios are involved:

  Scenario 1: ureg is empty

  • In this scenario, there is no valid data in ureg, indicating the start state of contiguous unaligned move-out. StoreUnAlign is called to write the data corresponding to Unified Buffer addresses 48 to 288 in srcReg to dstAddr, and cache the data corresponding to Unified Buffer addresses 288 to 320 in srcReg to ureg.
  • StoreUnAlignPost is called to perform post-processing for unaligned data move-out. The valid data corresponding to Unified Buffer addresses 288 to 320 cached in ureg is written to Unified Buffer.
  Figure 2 Unaligned data move-out (ureg is empty)

  

  Scenario 2: ureg is not empty

  • In this scenario, valid data exists in ureg. StoreUnAlign is called. The system concatenates the data corresponding to Unified Buffer addresses 32 to dstAddr in ureg with the data corresponding to Unified Buffer addresses dstAddr to 288 in srcReg, and writes the result to Unified Buffer address dstAddr. The data corresponding to Unified Buffer addresses 288 to 320 in srcReg is also cached to ureg. During contiguous unaligned data move-out, if ureg is not empty, the next StoreUnAlign operation will read the data cached in ureg by the current StoreAUnlign operation. Therefore, StoreUnAlignPost does not need to be called during the move-out, and it only needs to be called once after the move-out is complete, thereby optimizing the performance of unaligned data move-out.
  • StoreUnAlignPost is called to perform post-processing for unaligned data move-out. The valid data corresponding to Unified Buffer addresses 288 to 320 cached in ureg is written to Unified Buffer.
  Figure 3 Unaligned data move-out (ureg is not empty)

  

• Contiguous unaligned data move-out for MaskReg

  Data is moved out from the source register MaskReg (32 bytes) to Unified Buffer.

  • When the data corresponding to Unified Buffer addresses is of the b16 type, the hardware instruction extracts the least significant bit (LSB) from each two bits of data, packs the 32-byte data in MaskReg into 16-byte data, and writes the data to Unified Buffer. After the movement is complete, the Unified Buffer addresses are updated based on the 16-byte offset.
  • When the data corresponding to Unified Buffer addresses is of the b32 type, the hardware instruction extracts the least significant bit (LSB) from each four bits of data, packs the 32-byte data in MaskReg into 8-byte data, and writes the data to Unified Buffer. After the movement is complete, the Unified Buffer addresses are updated based on the 8-byte offset.

  Assume that the data corresponding to Unified Buffer addresses is of the b16 type, and data in two MaskReg registers is written to addresses dstAddr to 72. The unaligned data move-out process is as follows:

  • The 16-byte data (corresponding to Unified Buffer addresses dstAddr to 56) obtained after maskReg1 packing is cached to ureg.
  • The data in ureg and some of the 16-byte data (corresponding to Unified Buffer addresses 56 to 64) obtained after maskReg2 packing are concatenated, and the result is written to Unified Buffer addresses dstAddr to 64.
  • Some of the 16-byte data (corresponding to Unified Buffer addresses 64 to 72) obtained after maskReg2 packing is cached to ureg.
  • Post-processing for unaligned data move-out is performed. The data cached in ureg is written to Unified Buffer addresses 64 to 72. The process of contiguous unaligned data move-out is similar for MaskReg and RegTensor. You only need to call StoreUnAlignPost once after the move-out to optimize the performance of contiguous unaligned data move-out for MaskReg.
  Figure 4 Contiguous unaligned data move-out for MaskReg