Reducing the Number of Transfers in Non-contiguous Transfer Scenarios

[Priority] Medium

This performance optimization suggestion applies to the following product models:

Atlas 350 Accelerator Card

In non-contiguous transfer scenarios, you can use the Loop mode of the DataCopyPad API and the multi-dimensional data transfer API of DataCopy to reduce the number of transfers, thereby optimizing the transfer performance.

Using the Loop Mode to Reduce the Number of Non-contiguous Transfers

[Description] Based on the Normal or Compact mode, the DataCopyPad API can use the Loop mode to transfer two-dimensional data. Assume that eight 48-byte data blocks need to be transferred, as shown in the following figure.

[Negative Example] Call the transfer API multiple times (using DataCopyPad as an example).

__aicore__ inline void CopyIn3(){
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::DataCopyParams dataCopyParams;
    dataCopyParams.blockCount = 2;
    dataCopyParams.blockLen = 48;
    dataCopyParams.srcStride = 0;
    dataCopyParams.dstStride = 0;
    AscendC::DataCopyPadParams dataCopyPadParams;
    dataCopyPadParams.isPad = 0;
    dataCopyPadParams.leftPadding = 0;
    dataCopyPadParams.rightPadding = 0;
    dataCopyPadParams.paddingValue = 0;
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal, xGm, dataCopyParams, dataCopyPadParams);
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal[32], xGm[24], dataCopyParams, dataCopyPadParams);
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal[72], xGm[48], dataCopyParams, dataCopyPadParams);
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal[104], xGm[72], dataCopyParams, dataCopyPadParams);
    inQueueX.EnQue<T>(xLocal);
}

Figure 1 Calling the DataCopyPad API multiple times to transfer data

[Positive Example] Use the Loop mode for transfer.

__aicore__ inline void CopyIn3(){
    AscendC::LoopModeParams loopModeParams;
    loopModeParams.loop1Size = 2;
    loopModeParams.loop2Size = 2;
    loopModeParams.loop1SrcStride = 96;
    loopModeParams.loop1DstStride = 128;
    loopModeParams.loop2SrcStride = 192;
    loopModeParams.loop2DstStride = 288;
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::DataCopyParams dataCopyParams;
    dataCopyParams.blockCount = 2;
    dataCopyParams.blockLen = 48;
    dataCopyParams.srcStride = 0;
    dataCopyParams.dstStride = 0;
    AscendC::DataCopyPadParams dataCopyPadParams;
    dataCopyPadParams.isPad = 0;
    dataCopyPadParams.leftPadding = 0;
    dataCopyPadParams.rightPadding = 0;
    dataCopyPadParams.paddingValue = 0;
    AscendC::SetLoopModePara(loopModeParams, AscendC::DataCopyMVType::OUT_TO_UB);
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Compact>(xLocal, xGm, dataCopyParams, dataCopyPadParams);
    AscendC::ResetLoopModePara(AscendC::DataCopyMVType::OUT_TO_UB);
    inQueueX.EnQue<T>(xLocal);
}

Figure 2 Using the Loop mode for transfer

[Summary] When different sizes of padding data need to be inserted between data blocks, using the Loop mode for transfer instead of multiple DataCopyPad operations can reduce the number of transfer instructions and enhance performance.

Using Multi-dimensional Data Transfer to Reduce the Number of Non-contiguous Transfers

[Description] Assume that two 8-byte data blocks need to be transferred, as shown in the following figure.

Figure 3 Data before and after transfer

[Negative Example] Use DataCopyPad multiple times for transfer.

Figure 4 Calling DataCopyPad multiple times to transfer data

__aicore__ inline void CopyIn5(){
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::DataCopyParams dataCopyParams;
    dataCopyParams.blockCount = 1;
    dataCopyParams.blockLen = 8;
    dataCopyParams.srcStride = 0;
    dataCopyParams.dstStride = 0;
    AscendC::DataCopyPadParams dataCopyPadParams;
    dataCopyPadParams.isPad = 1;
    dataCopyPadParams.leftPadding = 5;
    dataCopyPadParams.rightPadding = 1;
    dataCopyPadParams.paddingValue = 0;
    // First transfer
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Normal>(xLocal, xGm, dataCopyParams, dataCopyPadParams);
    dataCopyPadParams.isPad = 1;
    dataCopyPadParams.leftPadding = 1;
    dataCopyPadParams.rightPadding = 5;
    dataCopyPadParams.paddingValue = 0;
    // Second transfer
    AscendC::DataCopyPad<T, AscendC::PaddingMode::Normal>(xLocal[8], xGm[2], dataCopyParams, dataCopyPadParams);
    inQueueX.EnQue<T>(xLocal);
}

[Positive Example] Use multi-dimensional data transfer.

The DataCopy API supports multi-dimensional data transfer on the Atlas 350 Accelerator Card. For details, see Multi-dimensional Data Transfer (ISASI). The following uses 2D data transfer as an example. The code is as follows:

__aicore__ inline void CopyIn6(){
    AscendC::LocalTensor<T> xLocal = inQueueX.AllocTensor<T>();
    AscendC::Duplicate<T>(xLocal, 0, count);
    AscendC::NdDmaLoopInfo<2> loopInfo{{1, 2}, {1, 4}, {2, 2}, {1, 1}, {1, 1}};
    AscendC::NdDmaParams<T, 2> params = {loopInfo, 0};
    AscendC::NdDmaDci();
    static constexpr AscendC::NdDmaConfig config = {false};
    AscendC::DataCopy<T, 2, config>(xLocal, xGm, params);
    inQueueX.EnQue<T>(xLocal);
}

Figure 5 Data before and after transfer

[Summary] In some scenarios, using multi-dimensional data transfer can reduce the number of transfer instructions, thereby enhancing performance.

Parent topic: Memory Access