Real-time ND2NZ Conversion During Data Transfer

Applicability

Function Usage

Supports ND-to-NZ format conversion during data transfer.

Prototype

• Global Memory -> Local Memory

  1 2	template <typename T> __aicore__ inline void DataCopy(const LocalTensor<T>& dst, const GlobalTensor<T>& src, const Nd2NzParams& intriParams)

  1 2 3

  // This function prototype is only supported by the Atlas 350 Accelerator Card. template <typename T, bool enableSmallC0 = false> __aicore__ inline void DataCopy(const LocalTensor<T>& dst, const GlobalTensor<T>& src, const Nd2NzParams& intriParams)

• Local Memory -> Local Memory

  1 2	template <typename T> __aicore__ inline void DataCopy(const LocalTensor<T>& dst, const LocalTensor<T>& src, const Nd2NzParams& intriParams)

Parameters

The following figure shows the ND2NZ conversion. The parameter settings in the example are described as follows:

• ndNum = 2 indicates that the number of ND matrices to transfer is 2 (ND matrix 1 is A1 to A2 + B1 to B2, and ND matrix 2 is C1 to C2 + D1 to D2).
• nValue = 2 indicates that the number of rows in the ND matrix is 2, that is, the height of the matrix is 2.
• dValue = 24 indicates that the number of columns in the ND matrix is 24, that is, the width of the matrix is 24 elements. If dValue is not 32-byte aligned, the insufficient part in the destination operand is padded with 0s. For example, the blank part in the data block where A2 is located is padded with 0s.
• srcNdMatrixStride = 144 indicates the stride between the start addresses of adjacent ND matrices, that is, the distance between A1 and C1, which is nine data blocks (9 × 16 = 144 elements).
• srcDValue = 48 indicates the number of elements in a row, that is, the distance between A1 and B1, which is three data blocks (3 × 16 = 48 elements).
• dstNzC0Stride = 11 indicates that after conversion from ND to NZ format, a single row in the source operand is split into multiple rows in the destination operand. For example, one row in src is divided into two separate rows A1 and A2 in dst. The stride between the start addresses of these rows corresponds to the stride between A1 and A2 in dst, which is 11 data blocks.
• dstNzNStride = 2 indicates the stride between the xth and (x+1)th rows of an ND matrix in src after conversion to the NZ format in dst. That is, the stride between A1 and B1 in dst is two data blocks.
• dstNzMatrixStride = 96 indicates the stride between the start addresses of the xth and (x+1)th ND matrices in dst, that is, the distance between A1 and C1, which is six data blocks (6 × 16 = 96 elements).

The following figure shows the ND2NZ conversion when enableSmallC0 is enabled.

Returns

None

Restrictions

For the Atlas inference product's AI Core, when the ND2NZ transfer API of the Global Memory -> Local Memory path is used, 8 KB UB space must be reserved as the temporary data storage area of the API.

Supported Paths and Data Types

The following data paths are expressed using logical positions TPosition, with the corresponding physical paths noted. For details about the mapping between TPosition and the physical memory, see Table 1.

Examples

For details about intriParams, see Figure 1.

// srcLocal is a LocalTensor of the half type in the L1 Buffer space, workLocal is a LocalTensor of the half type in the UB space, and srcGlobal is a GlobalTensor of the half type.
AscendC::Nd2NzParams intriParams{1, 32, 32, 0, 32, 32, 1, 0};
// Global Memory -> Local Memory
AscendC::DataCopy(srcLocal, srcGlobal, intriParams);
// Local Memory -> Local Memory
AscendC::DataCopy(srcLocal, workLocal, intriParams);

Result example:

Input (srcGlobal): [1 2 3 ... 1024]
Output (dstGlobal): [1 2 ... 15 16 33 34 ... 47 48 65 66 ... 79 80 97 98 ... 111 112 ... 1009 1010... 1023 1024]