Reducing Invalid Data Transfers in Non-Alignment Scenarios

[Priority] Medium

This performance optimization suggestion applies to the following models:

Atlas 350 Accelerator Card

[Description] In non-alignment data transfer scenarios, the Atlas 350 Accelerator Card provides the DataCopyPad API at the basic API level. This API supports two transfer modes: Normal and Compact. When multiple data blocks that are not 32-byte aligned are transferred, using the Compact mode can reduce the amount of invalid data transferred, thereby saving bandwidth.

Assume that three data blocks need to be transferred, each with a size of 48 bytes and a float data type. Except for these three 48-byte data blocks, all other data is invalid.

[Negative Example] Use the DataCopyPad API to transfer data in Normal mode.

__aicore__ inline void CopyIn(){
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::DataCopyParams dataCopyParams;
    dataCopyParams.blockCount = 3;
    dataCopyParams.blockLen = 48;
    dataCopyParams.srcStride = 0;
    dataCopyParams.dstStride = 0;
    AscendC::DataCopyPadParams dataCopyPadParams;
    dataCopyPadParams.isPad = 1;
    dataCopyPadParams.leftPadding = 0;
    dataCopyPadParams.rightPadding = 4;
    dataCopyPadParams.paddingValue = 0;
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Normal>(xLocal, xGm, dataCopyParams, dataCopyPadParams);
    inQueueX.EnQue<T>(xLocal);
}

The data in the UB after transfer is as follows:

[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 
 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0.,
 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0.....]

Figure 1 Transfer in Normal mode

As shown in the figure, each data block is 48 bytes and not 32-byte aligned. Therefore, 16-byte padding data needs to be inserted when each data block is transferred to achieve 32-byte alignment. Finally, 192-byte data is transferred to the UB, including 48-byte invalid data.

[Positive Example] Use the Compact mode for transfer optimization.

__aicore__ inline void CopyIn(){
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::DataCopyParams dataCopyParams;
    dataCopyParams.blockCount = 3;
    dataCopyParams.blockLen = 48;
    dataCopyParams.srcStride = 0;
    dataCopyParams.dstStride = 0;
    AscendC::DataCopyPadParams dataCopyPadParams;
    dataCopyPadParams.isPad = 1;
    dataCopyPadParams.leftPadding = 0;
    dataCopyPadParams.rightPadding = 4;
    dataCopyPadParams.paddingValue = 0;
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal, xGm, dataCopyParams, dataCopyPadParams);
    inQueueX.EnQue<T>(xLocal);
}

The data in the UB after transfer is as follows:

[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
 1., 1., 0., 0., 0., 0....]

Figure 2 Transfer in Compact mode

According to the diagram of transfer in Compact mode, 160-byte data is finally transferred, including 16-byte invalid data.

[Summary] Through comparison, it can be found that when multiple data blocks that are not 32-byte aligned are transferred, using the Compact mode can reduce the amount of invalid data transferred, thereby saving bandwidth.

Parent topic: Memory Access