Avoiding Bank Conflicts (Atlas 350 Accelerator Card)

To improve data access efficiency and throughput, the Unified Buffer adopts a bank structure (memory modules of the same size). The Unified Buffer has a total size of 256 KB and is divided into 16 banks. Each bank consists of 512 rows, with each row being 32 bytes long. These 16 banks are further organized into eight bank groups, with each bank group containing two banks. For example, bank7 and bank15 form a bank group.

Figure 1 Bank structure (the arrow direction indicates the memory layout sequence)

Each bank can independently read and write data, and multiple data requests can be processed at the same time. However, when a plurality of read/write operations attempt to access a same bank at the same time, due to a limitation of hardware resources, these operations need to wait in a queue, resulting in a bank conflict and performance deterioration.

Specifically, the Vector Unit can read or write a row of data from or into each bank group in each cycle (an instruction cycle). When multiple read/write operations attempt to access the same bank at the same time, the Vector Unit cannot process all requests in the same cycle. As a result, these requests wait in a queue. This queuing increases the data access latency and reduces the overall system performance.

Typical Scenarios of Bank Conflicts

Bank conflicts can be classified into the following three scenarios:

Read/Write conflict: A read operation and a write operation attempt to access the same bank at the same time.
Write/Write conflict: Multiple write operations attempt to access the same bank group at the same time.
Read/Read conflict: Two read operations attempt to access the same bank at the same time, or two or more read operations attempt to access the same bank group at the same time.

The following provides some specific examples. Assume that 0x10000 is on bank0 and 0x10020 is on bank1, as shown in the following figure.

Figure 2 Address allocation

Example of a read/write conflict

When the source operand src and destination operand dst of the Vector instruction are read and written to the same bank at the same time, a read/write conflict occurs. The details are as follows:

**Table 1** Example of a read/write conflict
No.	src addr	dst addr	bank	bank group	Conclusion
Example 1	0x10020	0x10000	bank_id0 != bank_id1	bank_group_id0 != bank_group_id1	The src and dst addresses are in bank0 and bank1, respectively. Therefore, no conflict occurs.
Example 2	0x10020	0x10120	bank_id0 == bank_id1	bank_group_id0 == bank_group_id1	The src and dst addresses are in bank0. Therefore, a conflict occurs.

Example of a write/write conflict

When eight data blocks (block0 to block7) corresponding to the destination operand dst of the Vector instruction are written to the same bank group, a write/write conflict occurs. The details are as follows:

**Table 2** Example of a write/write conflict
No.	dst addr	blk_stride	block0_addr	block1_addr	block2_addr	...	Conclusion
Example 1	0x10000	8	0x10000	0x10100	0x10200	...	All eight data blocks are in the same bank group. Therefore, conflicts occur. One repeat is written in eight cycles.

Read/Read conflict

When two source operands of the Vector instruction are read to the same bank at the same time, a read/read conflict occurs. The analysis is as follows:

**Table 3** Example of a read/read conflict with two source addresses
No.	src0 addr	src1 addr	bank	bank group	Conclusion
Example 1	0x10000	0x10100	bank_id0 == bank_id1	bank_group_id0 == bank_group_id1	Conflict
Example 2	0x10000	0x10020	bank_id0 != bank_id1	bank_group_id0 != bank_group_id1	No conflict

When the eight data blocks (block0 to block7) corresponding to a source operand of the Vector instruction are read to the same bank, a read/read conflict occurs. The analysis is as follows:

**Table 4** Example of a read/read conflict with a single source address
No.	src addr	blk_stride	block0_addr	block1_addr	block2_addr	...	Conclusion
Example 1	0x10000	8	0x10000	0x10100	0x10200	...	All eight data blocks are in the same bank. Therefore, conflicts occur. One repeat is read in eight cycles.

How to Avoid Bank Conflicts

There are two methods to avoid bank conflicts: optimizing the computation logic and optimizing address allocation.

Optimizing the computation logic

Add 1 to each element of an input with the data type float and shape (16, 64). By changing the computation logic from column-wise to row-wise, conflicts within the same repeat can be avoided. The following table compares the implementation solutions.

Implementation

Original Implementation

Optimized Implementation

Implementation method

Column-wise computation: The eight data blocks input in the same repeat are in the same bank, causing a read/read conflict.

Row-wise computation: The eight data blocks input in the same repeat are not in the same bank, avoiding a read/read conflict within the same repeat.

Diagram

Sample code

uint64_t mask = 64;
AscendC::UnaryRepeatParams params;
params.dstBlkStride = 8;
params.srcBlkStride = 8;
for(uint16_t i = 0; i < 8; ++i){
    AscendC::Adds(dst[i * 8], src[i * 8], 1, mask, 1, params);
}

uint64_t mask = 64;
AscendC::UnaryRepeatParams params;
params.dstBlkStride = 1;
params.srcBlkStride = 1;
for(uint16_t i = 0; i < 8; ++i){
    AscendC::Adds(dst[i * 64], src[i * 64], 1, mask, 1, params);
}

Optimizing address allocation

Implement the addition of 4096 consecutive float elements (z = x + y). By allocating more memory, ensure that x/y and z do not appear in the same bank at the same time within a repeat.

The implementation solutions are compared as follows:

Implementation

Original Implementation

Optimized Implementation

Implementation method

No address optimization is performed. InitBuffer is directly used to allocate memory. The addresses of each tensor are as follows:

x: The start address is 0x00000, and the tensor length is 4096 x sizeof(float) bytes.

y: The start address is 0x04000, and the tensor length is 4096 x sizeof(float) bytes.

z: The start address is 0x08000, and the tensor length is 4096 x sizeof(float) bytes.

In a repeat, x and y read the same bank group at the same time, and x/y and z read and write the same bank at the same time.

Optimize the address. When InitBuffer is used to allocate memory, properly allocate more memory. The addresses of each tensor are as follows:

x: The start address is 0x00000, and the tensor length is 4096 x sizeof(float) bytes.

y: The start address is 0x04000, and the tensor length is (8 x 16 x 1024 – (4096 x sizeof(float))) bytes.

z: The start address is 0x20000, and the tensor length is 4096 x sizeof(float) bytes.

Allocate more space for y to ensure that z does not fall into the same bank as x or y.

Diagram

Sample code

pipe.InitBuffer(inQueueX, 1, 4096 * sizeof(float));
pipe.InitBuffer(inQueueY, 1, 4096 * sizeof(float));
pipe.InitBuffer(outQueueZ, 1, 4096 * sizeof(float));

constexpr int32_t TOTAL_LENGTH = 1024 * 4;
constexpr int32_t BUFFER_NUM = 1;
constexpr int32_t BANKGROUP_SIZE  =  1024 * 128; 
...
pipe.InitBuffer(inQueueX, BUFFER_NUM, TOTAL_LENGTH * sizeof(float));
pipe.InitBuffer(inQueueY, BUFFER_NUM, BANKGROUP_SIZE - TOTAL_LENGTH * sizeof(float));
pipe.InitBuffer(outQueueZ, BUFFER_NUM, TOTAL_LENGTH * sizeof(float));

Parent topic: Avoiding Bank Conflicts in the Unified Buffer