Avoiding Bank Conflicts in the Unified Buffer

[Priority] High

This performance optimization guide applies to the following product models:

  • Atlas A3 training products / Atlas A3 inference products
  • Atlas A2 training products / Atlas A2 inference products

[Description] To improve data access efficiency and throughput, the Unified Buffer adopts a bank (memory modules of the same size) structure. The Unified Buffer has a total size of 192 KB and is divided into 48 banks. Each bank consists of 128 rows, with each row being 32 bytes long. These 48 banks are further organized into 16 bank groups, with each bank group containing three banks. For example, bank 15, bank 31, and bank 47 form a bank group.

Figure 1 Bank structure (the arrow direction indicates the memory layout sequence)

Each bank can independently read and write data, and multiple data requests can be processed at the same time. However, when a plurality of read/write operations attempt to access a same bank or bank group at the same time, due to a limitation of hardware resources, these operations need to wait in a queue, resulting in a bank conflict and performance deterioration.

Specifically, the Vector Unit can read or write a row of data from or into each bank group in each cycle (an instruction cycle). If multiple operations in the same API attempt to access the same bank or bank group at the same time, the Vector Unit cannot process all requests in the same cycle. As a result, these requests wait in a queue. This queuing increases the data access latency and reduces the overall system performance.

Typical Scenarios of Bank Conflicts

Bank conflicts can be classified into the following three scenarios:

The following provides some specific examples. Assume that 0x10000 is on bank 16, 0x10020 is on bank 17, and 0x20020 is on bank 33, as shown in the following figure.

Figure 2 Address allocation

The msProf tool can be used to collect profile data related to the resource conflict ratio.

For details about how to use the tool, see msProf (Operator Tuning). For details about the profile data file, see ResourceConflictRatio (Resource Conflict Ratio).

How to Avoid Bank Conflicts

There are two methods to avoid bank conflicts: optimizing the computation logic and optimizing address allocation.