LoadDataWithTranspose

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	√
Atlas inference product AI Core	x
Atlas inference product Vector Core	x
Atlas training product	x

Function Usage

Loads 2D data with transposing from A1/B1 to A2/B2.

The following uses an example to describe the API functions and key parameters. In the following figure, an N shape or a Z shape represents a fractal.

For uint8_t and int8_t, each iteration processes 32 × 32 × 1 byte of data, which can be divided into 2 fractals (each fractal is 512 bytes). In each iteration, two consecutive 16 × 32 fractals in the source operand are combined into one 32 × 32 square matrix. A transpose operation is then performed on this square matrix. After transposition, the matrix is split back into two 16 × 32 fractals. These fractals can be arranged in different layouts according to parameters such as the fractal spacing in the destination operand.
The following figure shows an example.
- A total of 3,072 bytes of data needs to be processed. Each iteration processes 32 × 32 × 1 byte of data, so 3 iterations are required to complete the processing. Therefore, repeatTime = 3.
- srcStride = 1: The stride between the start addresses of two adjacent square matrices of the source operand in adjacent iterations is 1 (unit: 32 × 32 × 1 byte). The unit is the size of the square matrix after combination.
- dstGap = 1: The gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand is 1 (unit: 512 bytes).
- dstFracGap = 0: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 0 (unit: 512 bytes).
The following figure shows an example.
- The descriptions of repeatTime and srcStride are the same as those in the preceding figure.
- dstGap = 0: There is no gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand.
- dstFracGap = 2: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 2 (unit: 512 bytes).

For half and bfloat16_t, each iteration processes 16 × 16 × 2 bytes of data, which corresponds to one fractal (each fractal is 512 bytes). In each iteration, a single 16 × 16 fractal in the source operand is transposed.
- A total of 1,536 bytes of data needs to be processed. Each iteration processes 16 × 16 × 2 bytes of data, so 3 iterations are required to complete the processing. Therefore, repeatTime = 3.
- srcStride = 1: The stride between the start addresses of two adjacent square matrices of the source operand in adjacent iterations is 1 (unit: 16 × 16 × 2 bytes).
- dstGap = 0: There is no gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand.
- In this scenario, the fractal is a square matrix. Each iteration processes a fractal, and there is no interval between fractals in the iteration. Therefore, this parameter is invalid.

For float, int32_t, and uint32_t, each iteration processes 16 × 16 × 4 bytes of data, which can be divided into 2 fractals (each fractal is 512 bytes). In each iteration, two consecutive 16 × 8 fractals in the source operand are combined into one 16 × 16 square matrix. A transpose operation is then performed on this square matrix. After transposition, the matrix is split back into two 16 × 8 fractals. These fractals can be arranged in different layouts according to parameters such as the fractal spacing in the destination operand.
The following figure shows an example.
- A total of 3,072 bytes of data needs to be processed. Each iteration processes 16 × 16 × 4 bytes of data, so 3 iterations are required to complete the processing. Therefore, repeatTime = 3.
- srcStride = 1: The stride between the start addresses of two adjacent square matrices of the source operand in adjacent iterations is 1 (unit: 16 × 16 × 4 bytes). The unit is the size of the square matrix after combination.
- dstGap = 1: The gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand is 1 (unit: 512 bytes).
- dstFracGap = 0: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 0 (unit: 512 bytes).
The following figure shows an example.
- The descriptions of repeatTime and srcStride are the same as those in the preceding figure.
- dstGap = 0: There is no gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand.
- dstFracGap = 2: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 2 (unit: 512 bytes).
For int4b_t, each iteration processes 64 × 64 × 0.5 bytes of data, which can be divided into 4 fractals (each fractal is 512 bytes). In each iteration, four consecutive 16 × 64 fractals in the source operand are combined into one 64 × 64 square matrix. A transpose operation is then performed on this square matrix. After transposition, the matrix is split back into four 16 × 64 fractals. These fractals can be arranged in different layouts according to parameters such as the fractal spacing in the destination operand.
The int4b_t data type requires two numbers to be combined into a number of type int8_t or uint8_t. The combination rule is as follows.
The following figure shows an example.
- A total of 6,144 bytes of data needs to be processed. Each iteration processes 64 × 64 × 0.5 bytes of data, so 3 iterations are required to complete the processing. Therefore, repeatTime = 3.
- srcStride = 1: The stride between the start addresses of two adjacent square matrices of the source operand in adjacent iterations is 1 (unit: 64 × 64 × 0.5 bytes). The unit is the size of the square matrix after combination.
- dstGap = 1: The gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand is 1 (unit: 512 bytes).
- dstFracGap = 0: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 0 (unit: 512 bytes).
The following figure shows an example.
- The descriptions of repeatTime and srcStride are the same as those in the preceding figure.
- dstGap = 0: There is no gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand.
- dstFracGap = 2: The gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration is 2 (unit: 512 bytes).

Prototype

template <typename T>
__aicore__ inline void LoadDataWithTranspose(const LocalTensor<T>& dst, const LocalTensor<T>& src, const LoadData2dTransposeParams& loadDataParams)

// This function prototype is only supported by the Atlas 350 Accelerator Card.
template <typename T>
__aicore__ inline void LoadDataWithTranspose(const LocalTensor<T>& dst, const LocalTensor<T>& src, const LoadData2dTransposeParamsV2& loadDataParams)

Parameters

**Table 1** Template parameters
Parameter	Description
T	For the Atlas A2 training product/Atlas A2 inference product, the supported data types are int4b_t, int8_t, uint8_t, half, bfloat16_t, float, int32_t, and uint32_t. For int4b_t, data can be transferred only from B1 to B2. For the Atlas A3 training product/Atlas A3 inference product, the supported data types are int4b_t, int8_t, uint8_t, half, bfloat16_t, float, int32_t, and uint32_t. For int4b_t, data can be transferred only from B1 to B2. For the Atlas 200I/500 A2 inference product, the supported data types are int4b_t, uint8_t, int8_t, uint16_t, int16_t, half, bfloat16_t, uint32_t, int32_t, and float. For the Atlas 350 Accelerator Card, the supported data types are int8_t, uint8_t, half, bfloat16_t, float, int32_t, and uint32_t. The int4b_t type is supported only when TPosition of the LocalTensor is B2.

**Table 2** Parameters
Parameter	Input/Output	Description
dst	Output	Destination operand for the result matrix, which is of the LocalTensor type. For the Atlas A2 training product/Atlas A2 inference product, TPosition can only be A2 or B2. For the Atlas A3 training product/Atlas A3 inference product, TPosition can only be A2 or B2. For the Atlas 200I/500 A2 inference product, TPosition can only be A2 or B2. For the Atlas 350 Accelerator Card, TPosition can only be B2. The start address of LocalTensor must be 512-byte aligned. The data type must be the same as that of src.
src	Input	Source operand, which is of the LocalTensor type. For the Atlas A2 training product/Atlas A2 inference product, TPosition can only be A1 or B1. For the Atlas A3 training product/Atlas A3 inference product, TPosition can only be A1 or B1. For the Atlas 200I/500 A2 inference product, TPosition can only be A1 or B1. For the Atlas 350 Accelerator Card, TPosition can only be B1. The start address of LocalTensor must be 32-byte aligned. The data type must be the same as that of dst.
loadDataParams	Input	Parameters related to LoadDataWithTranspose, which is of the LoadData2dTransposeParams type. For details, see ${INSTALL_DIR}/include/ascendc/basic_api/interface/kernel_struct_mm.h. Replace ${INSTALL_DIR} with the actual path for storing files after the CANN software is installed. For details about the parameter description, see Table 3.
loadDataParams	Input	Parameters related to LoadDataWithTranspose, which is of the LoadData2dTransposeParamsV2 type. For details about the parameter description, see Table 4.

**Table 3** Parameters in the LoadData2dTransposeParams structure
Parameter	Input/Output	Description
startIndex	Input	Square matrix ID, indicating the sequence number of the square matrix of the source operand from which the movement starts (0 indicates the first square matrix of the source operand). Value range: startIndex ∈ [0, 65535]. Default value: 0. For example, if the source operand has 20 fractals whose size is 16 × 8 × 4 bytes (data type: float) and startIndex is 1, the transfer start position is the second square matrix, that is, the third and fourth fractals are transposed from the source operand to the destination operand (the first and second fractals form the first square matrix, and the third and fourth fractals form the second square matrix).
repeatTimes	Input	Number of iterations. For uint8_t and int8_t, 32 × 32 × 1 byte of data is processed in each iteration. For half and bfloat16_t, 16 × 16 × 2 bytes of data is processed in each iteration. For float, int32_t, and uint32_t, 16 × 16 × 4 bytes of data is processed in each iteration. For int4b_t, 16 × 64 × 0.5 bytes of data is processed in each iteration. Value range: repeatTimes ∈ [0, 255]. Default value: 0.
srcStride	Input	In adjacent iterations, the interval between the start addresses of consecutive fractal matrices in the source operand. The unit here is actually the size of the combined square matrix. For uint8_t and int8_t, the unit is 32 × 32 × 1 byte. For half and bfloat16_t, the unit is 16 × 16 × 2 bytes. For float, int32_t, and uint32_t, the unit is 16 × 16 × 4 bytes. For int4b_t, 16 × 64 × 0.5 bytes of data is processed in each iteration. Value range: srcStride ∈ [0, 65535]. Default value: 0.
dstGap	Input	Gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand (unit: 512 bytes). Value range: dstGap ∈ [0, 65535]. Default value: 0.
dstFracGap	Input	Gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration (unit: 512 bytes). This parameter is valid only when the data type is float, int32_t, uint32_t, uint8_t, int8_t, or int4b_t. Value range: dstFracGap ∈ [0, 65535]. Default value: 0.
addrMode	Input	Address update mode. Default value: false. true: Decrement. In each iteration, the new address is obtained by subtracting srcStride from the previous address. false: Increment. In each iteration, the new address is obtained by adding srcStride to the previous address.

**Table 4** Parameters in the LoadData2dTransposeParamsV2 structure
Parameter	Input/Output	Description
startIndex	Input	Square matrix ID, with the transfer start position defined as the nth fractal in the source operand. Value range: startIndex ∈ [0, 65535]. Default value: 0.
repeatTimes	Input	Number of iterations. For int4b_t, four fractals are processed per iteration, with each fractal containing 16 × 64 × 0.5 bytes of data. For uint8_t and int8_t, two fractals are processed per iteration, with each fractal containing 16 × 32 × 1 byte of data. For half and bfloat16_t, one fractal is processed per iteration, with each fractal containing 16 × 16 × 2 bytes of data. For int32_t, uint32_t, and float, four fractals are processed per iteration, with each fractal containing 16 × 8 × 4 bytes of data. Value range: repeatTimes ∈ [1, 255].
srcStride	Input	In adjacent iterations, the interval between the start addresses of fractals of the source operand. The unit is 512 bytes per fractal. Value range: srcStride ∈ [0, 65535]. Default value: 0.
dstGap	Input	Gap between the end address of the first fractal in the previous iteration and the start address of the first fractal in the next iteration of the destination operand (unit: 512 bytes). Value range: dstGap ∈ [0, 65535]. Default value: 0.
dstFracGap	Input	Gap between the end address of the previous fractal and the start address of the next fractal of the destination operand in each iteration (unit: 512 bytes). This parameter is valid only when the data type is float, int32_t, uint32_t, uint8_t, int8_t, or int4b_t.
srcFracGap	Input	Gap between the end address of the previous fractal and the start address of the next fractal of the source operand in each iteration (unit: 512 bytes). This parameter is valid only when the data type is float, int32_t, uint32_t, uint8_t, int8_t, or int4b_t.
addrMode	Input	Address update mode. Default value: false. true: Decrement. In each iteration, the new address is obtained by subtracting srcStride from the previous address. false: Increment. In each iteration, the new address is obtained by adding srcStride to the previous address.

Restrictions

If repeat is set to 0, no movement operation is performed.
Developers need to ensure that the fractals after the transposition of the target operand do not overlap.
For details about the operand address alignment requirements, see General Address Alignment Restrictions.
For the following models, LoadData2dTransposeParamsV2 is recommended, which supports a finer data transfer granularity.
- Atlas 350 Accelerator Card

Examples

Example 1: The input matrix a is of the int8_t type, and its shape is [16,32]. The input matrix b is of the int8_t type, and its shape is [32,64]. The output matrix c is of the int32_t type. Matrix a is not transposed from A1 to A2, matrix b is transposed from B1 to B2, and then Mmad and Fixpipe are performed.

uint16_t m = 16, k = 32, n = 64;
uint8_t nBlockSize = 16;
uint16_t c0Size = 16;
uint16_t nBlockSize = 32;
AscendC::LoadData2dTransposeParams loadDataParams;
loadDataParams.startIndex = 0;
loadDataParams.repeatTimes = n / nBlockSize;
loadDataParams.srcStride = 1;
loadDataParams.dstGap = 1;
loadDataParams.dstFracGap = 0;
AscendC::LoadDataWithTranspose(b2Local, b1Local, loadDataParams);

Example 2: In this example, the input matrix a is of the half type and the shape is [16,32], the input matrix b is of the half type and the shape is [32,32], and the output matrix c is of the float type. Matrix a is not transposed from A1 to A2, matrix b is transposed from B1 to B2, and then Mmad and Fixpipe are performed.

AscendC::LocalTensor<half> b1Local = inQueueB1.DeQue<half>();
AscendC::LocalTensor<half> b2Local = inQueueB2.AllocTensor<half>();

uint16_t m = 16, k = 32, n = 32;
uint32_t nBlockSize = 16;
AscendC::LoadData2dTransposeParams loadDataParams;
loadDataParams.startIndex = 0;
loadDataParams.repeatTimes = k / nBlockSize;
loadDataParams.srcStride = 1;
loadDataParams.dstGap = 1;
for (int i = 0; i < (n / nBlockSize); ++i) {
    AscendC::LoadDataWithTranspose(b2Local[i * 16 * nBlockSize], b1Local[i * k * nBlockSize], loadDataParams);
}

inQueueB1.FreeTensor(b1Local);
inQueueB2.EnQue<half>(b2Local);

Example 3: In this example, the input matrix a is of the float type and the shape is [16,16], the input matrix b is of the float type and the shape is [16,32], and the output matrix c is of the float type. Matrix a is not transposed from A1 to A2, matrix b is transposed from B1 to B2, and then Mmad and Fixpipe are performed.

uint32_t m = 16, k = 16, n = 32;
uint32_t nBlockSize = 16;
AscendC::LocalTensor<half> b1Local = inQueueB1.DeQue<half>();
AscendC::LocalTensor<half> b2Local = inQueueB2.AllocTensor<half>();

AscendC::LoadData2dTransposeParams loadDataParams;
loadDataParams.startIndex = 0;

loadDataParams.repeatTimes = n / nBlockSize;
loadDataParams.srcStride = 1;
loadDataParams.dstGap = 0;
loadDataParams.dstFracGap = n / nBlockSize - 1;
AscendC::LoadDataWithTranspose(b2Local, b1Local, loadDataParams);
inQueueB1.FreeTensor(b1Local);
inQueueB2.EnQue<half>(b2Local);

Example 4: In this example, the LoadData2dTransposeParamsV2 structure is used. The input matrix a is of the int8_t type with a shape of [128, 128] and follows the NZ data format. The input matrix b is of the int8_t type with a shape of [128, 256] and also follows the NZ format. The output matrix c is of the float type. Matrix a is transferred from A1 to A2 without transposition, while matrix b is transferred from B1 to B2 with transposition. This example only demonstrates the API calling process; other computation and data transfer details are not for reference.

uint32 m = 256;
uint32 n = 256;
uint32 k = 128;
pipe = tpipe;
TQue<TPosition::B1, 1> qidB1_;
TQue<TPosition::B2, 1> qidB2_;
uint32 m = 128;
pipe->InitBuffer(qidB1_, 1, n * k * sizeof(int8_t));
pipe->InitBuffer(qidB2_, 1, n * k * sizeof(int8_t));
auto rightMatrix = qidB1_.template DeQue<int8_t>();
LocalTensor<int8_t> b2 = qidB2_.AllocTensor<int8_t>();
uint16_t fracNum = 2;
uint16_t kStep = CeilDiv(kLength, 16);
uint16_t nStep = CeilDiv(nLength, 32);
for (uint16_t i = 0; i < nStep; i ++) {
    LoadData2dTransposeParamsV2 loadDataParams;
    loadDataParams.startIndex = i * kStep;
    loadDataParams.repeatTimes = kStep / 2;
    loadDataParams.srcStride = 2;
    loadDataParams.dstGap = nStep*2 - 1;
    LoadDataWithTranspose(b2[1024*i], rightMatrix, loadDataParams);
}
qidB2_.EnQue(b2);
qidB1_.FreeTensor(rightMatrix);

Parent topic: Data Movement