Abstract Hardware Architecture
SIMT programming on the AI Core allows instructions to independently address data, thereby supporting thread-level parallel compute. This mode is especially suitable for scenarios such as discrete access and complex control logic. Currently, only Atlas 350 Accelerator Card is supported.
SIMT programming can simplify the development of complex operators and irregular control flows. It effectively mitigates divergent computations involving branches and controls program complexity. At the system level, this helps improve hardware utilization and energy efficiency. As shown in Figure 1, an AI processor has multiple Vector Cores. Each Vector Core contains a compute unit, shared memory (Unified Buffer), registers, and stack space. The Global Memory outside the cores is shared by all Vector Cores.
The following describes the hardware resources involved in SIMT multi-thread computation:
- Each thread has its own register and stack space for storing local data. The number of registers is affected by the number of threads in a thread block. The more threads there are, the fewer registers each thread has.
- A part of the Unified Buffer serves as the shared memory for all threads within a thread block, enabling data exchange between threads in the thread block. Another part is used as the data cache for reading the Global Memory.
- In SIMT mode, when data in the Global Memory is read, the data is transferred through the data cache unit. The data flow goes from the global memory to the data cache, and then from the data cache to the register. The data cache is a part of the Unified Buffer and has a maximum capacity of 128 KB and a minimum capacity of 32 KB. The actual size is determined by the user.
If you are not familiar with concepts such as threads and thread blocks, you are advised to read Thread Architecture to learn more about the SIMT thread architecture. You can also refer to Memory Hierarchy to learn how to configure the division of the data cache and Unified Buffer.