NPU Architecture Version 351x

Hardware Architecture

As shown in the following figure, the AI Core in this architecture consists of two independent cores: AIC and AIV, which are used for cube computation and vector computation, respectively. The core ratio of AIC and AIV cores is 1:2. Each core has its own Scalar Unit and can independently load its own code segments.

The architecture has the following key features:

• Added the data paths from the L0C Buffer to the Unified Buffer and between the Unified Buffer and L1 Buffer.
• Deleted the data paths from the global memory to the L0A Buffer and from the global memory to the L0B Buffer.
• Deleted the data path from the L1 Buffer to the global memory.
• Used SSBuffer for inter-core communication between the AIC and AIV.
• Added the SIMD register file storage hierarchy. In SIMD programs, data is transferred from the Unified Buffer to the register for computation. The intermediate results generated can be directly computed in the register without being transferred back to the Unified Buffer.
• Added SIMT-related hardware units. The following table describes the SIMT-related hardware units.

  SIMT Hardware Unit	Description
  SIMT DCache	The SIMT accesses the global memory through the SIMT DCache. The SIMT supports a maximum of 128 KB data cache. The data cache directly reuses the Unified Buffer as the cache line. All external memory accesses of the SIMT are performed at a granularity of 128 bytes.
  Warp scheduler	This unit implements multithread scheduling of the hardware. Each AIV has four warp schedulers.
  SIMT register file	128 KB ultra-large-capacity register for SIMT applications. The number of registers available for each thread depends on the number of threads. The mapping is as follows: 1025–2048 threads: 16 registers 513–1024 threads: 32 registers 257–512 threads: 64 registers 1–256 threads: 127 registers

Compute Units

Separated deployment of the Cube Unit and Vector Unit

In this architecture, the Cube Unit and Vector Unit are deployed on the AIC and AIV cores, respectively. Each core has its own Scalar Unit and can independently load its own code segments.

Vector Unit

• The Vector Unit supports the U8, U16, U32, S8, S16, S32, BF16, FP16, and FP32 data types.
• The Vector Unit can process 256-byte data per cycle.
• The data processed by the Vector Unit comes from the register.
• In this architecture version, the mask value passed to the high-dimensional sharding API is converted into MaskReg and then passed to the Vector Unit. In NPU architecture version 220x, the mask value is read from a special mask register.
  Figure 1 High-dimensional sharding in the NPU architecture version 220
  
  Figure 2 High-dimensional sharding in this architecture version
  

Cube Unit

• The Cube Unit supports FP32, FP16, BF16, HiF8, FP8_E4M3, U8, and S8. A multiplication of the 16×16 and 16×16 cubes of the float16 data type is completed in one cycle. If the data type is int8_t, a multiplication of the 16×32 and 32×16 cubes is completed in one cycle.
• The Cube Unit can access the L0A Buffer, L0B Buffer, and L0C Buffer. The L0A Buffer stores the left cube, the L0B Buffer stores the right cube, and the L0C Buffer stores the cube multiplication result and intermediate result.

Scalar Unit

• The Scalar Unit supports the U16, S16, U32, S32, U64, S64, and FP64 data types.
• In the Regbase architecture, the Aux Scalar Unit independently processes scalar computation within the SIMD_VF function, while the Scalar Unit processes scalar computation outside the SIMD_VF function.

Storage Units

Obtaining the memory size of a storage unit

You can query the memory size of each storage unit by calling the platform information acquisition API.

Minimum access granularities (alignment requirements) of each storage unit

Recommended data layout formats for each storage unit

• The following fractal formats are recommended for the L0A Buffer, L0B Buffer, and L0C Buffer:
  • L0A Buffer: FRACTAL_NZ (Due to hardware structure changes, the fractal format of the L0A Buffer is changed to NZ in this architecture.)
  • L0B Buffer: FRACTAL_ZN
  • L0C Buffer: FRACTAL_NZ

  These formats are optimized for compute-intensive tasks such as cube multiplication, significantly improving the compute efficiency.

• The FRACTAL_NZ format is recommended for the L1 Buffer. When the L1 Buffer uses the NZ format, the format conversion overhead is reduced when data is moved to the L0A/L0B Buffer (which needs to be converted to the ZN format).
• The Unified Buffer has no requirements for the data format.

Access conflicts of storage units

The following figure shows the UB structure in this NPU architecture. When multiple operations attempt to access the same bank or bank group of the Unified Buffer at the same time, bank conflicts may occur, including the read-write conflict, write-write conflict, and read-read conflict. As a result, the accesses queue and the performance deteriorates. In NPU architecture version 220x, a bank group has only one group of read and write ports, and at most one read or one write operation can be completed in one cycle. In this NPU architecture, each bank group has two groups of read and write ports, and at most two read operations and no write operations or one read operation and one write operation can be performed at the same time. The related read and write constraints are as follows:

• Read/Write conflict: A read operation and a write operation attempt to access the same bank at the same time.
• Write/Write conflict: Multiple write operations attempt to access the same bank group at the same time.
• Read/Read conflict: Two read operations attempt to access the same bank at the same time, or two or more read operations attempt to access the same bank group at the same time.

Register

• RegTensor

  RegTensor is used to store Reg vector computation data. The bit width of RegTensor is the vector length (VL, 256 bytes).

• MaskReg

  MaskReg is used to indicate which elements are involved in the computation. The width is one-eighth (VL/8) of RegTensor.

• UnalignRegForLoad & UnalignRegForStore

  UnalignRegForLoad and UnalignRegForStore serve as buffers to optimize the overhead of contiguous unaligned address accesses between the UB and RegTensor. Before reading an unaligned address, UnalignRegForLoad and UnalignRegForStore should be initialized using the LoadUnAlignPre API, and then the LoadUnAlign API should be used. When writing an unaligned address, use the StoreUnAlign API first, and then use the StoreUnAlignPost API for post-processing.

• AddrReg

  AddrReg indicates an address register used to store address offsets. AddrReg is initialized through the CreateAddrReg API and then used to store address offsets in the loop. AddrReg is incremented based on the configured stride in each loop.

DMA Units

Alignment requirements during movement

The size of the data to be moved is subject to certain requirements because the moved data is used for computation. The size of the data moved to the Unified Buffer must be aligned based on the data block size. The data moved to other storage units must be aligned based on the fractal requirements. For example, when data is moved from the L1 Buffer to the L0A Buffer, the data format needs to be converted from NZ to ZN. The size of the data to be moved must be aligned based on the fractal size. If the remaining size of the L1 Buffer is less than one fractal, an exception will occur during hardware execution.

MTE hard channel

• A hard channel between the Unified Buffer and the L1 Buffer is added for the AIV.
• The loop mode for data movement from the global memory to the Unified Buffer and from the Unified Buffer to the global memory is supported. In loop mode, each loop can be in normal or compact mode. For details about the normal and compact modes, see DataCopyPad(ISASI).
  • Movement in normal mode in a single loop

    If the length of a single data block is 32-byte aligned, padding is not required. Multiple data blocks can be moved by repeating the movement for multiple times.

    

    If the length of a single data block is not 32-byte aligned, padding needs to be inserted after each data block to align the length to 32 bytes before the movement.

    

  • Movement in compact mode in a single loop

    In compact mode, a group of data blocks can be moved at a time. If the total length of these data blocks is 32-byte aligned, padding does not need to be inserted at the end.

    

    If the length of a group of data blocks is not 32-byte aligned, padding needs to be inserted after the group of data blocks to align the total length to 32 bytes.

    

FixPipe hardware acceleration

FixPipe is an acceleration module that hardens typical operations on the NPU. It is located inside the AIC and works with the Cube Unit to complete on-the-fly computation. Its main functions are as follows:

• Quantization and dequantization: including S4, S8, S32, FP16, FP32, FP8_E4M3, HiF8, and BF16.
• ReLU functions, including typical activation functions such as ReLU, PReLU, and Leaky ReLU.
• Data format conversion, including:
  • Fractal size conversion through Channel Merge and Channel Split, ensuring that the fractal output to the L1 Buffer/GM meets the requirements.
  • NZ2ND and NZ2DN conversion of L0C source.

Channel Merge supports the S8, U8, S4, and U4 data types, while Channel Split supports the FP32 data type.

• Channel Merge (S8 and U8 data types)

  For the target data type converted to S8 or U8, the fractal cube is converted from 16×16 to 16×32 through hardware. If the number of output channels (N) is an even multiple of 16, every two adjacent 16×16 fractal cubes in the N direction are merged into one 16×32 fractal cube. If the value of N is an odd multiple of 16, channels [1, N–16] are merged; the final 16 channels are not merged.

  As shown below, the target data type is S8, M is 32, and N is 48. First, the 16×16 fractal cubes in the first two columns are merged into one 16×32 cube, and then the remaining 16×16 fractal cubes are directly moved to the L1 Buffer.

  

• Channel Merge (S4 and U4 data types)

  For the target data type converted to S4 or U4, the fractal cube is converted from 16×16 to 16×32 through hardware. If the number of output channels (N) is a multiple of 64, every four adjacent 16×16 fractal cubes in the N direction are merged into a single 16×64 fractal cube.

  For example, the target data type is S4, M is 32, and N is 64. First, the 16×16 fractal cubes in the first row are merged into a 16×64 cube, and then the 16×16 fractal cubes in the second row are also merged.

  In this case, N must be a multiple of 64.

  

• FP32 Channel Split:

  For the target type FP32, the fractal cube can be converted from 16×16 to 16×8 through hardware. If Channel Split is enabled, each 16×16 fractal cube is split into two 16×8 fractal cubes.

  As shown in the following figure, the target data type is FP32, M is 64, and N is 32. The cube is split into sixteen 16×8 fractals.

  

AIC-AIV Inter-Core Communication

This architecture supports inter-core communication with the AIC:AIV ratio of 1:1 or 1:2. The inter-core communication is implemented through the SSBuf, which is different from the NPU220 architecture where the inter-core communication is implemented through the global memory.

Synchronization Control

• Intra-core synchronization

  The execution units (such as the MTE2 and Vector Unit) within the AI Core operate asynchronously and in parallel. As a result, data dependencies may arise when reading data from or writing data to the local memory (such as the Unified Buffer). To ensure data consistency and computational accuracy, synchronization control is required to coordinate the timing of operations.

  For example, in the process where the MTE2 moves data from the GM to the UB for Abs computation by the Vector Unit and then moves the data back to the GM, the following synchronization conditions must be met:

  1. Data movement and computation sequence
     • Start Abs computation by the Vector Unit after data is moved from the GM to the UB (to avoid data loss caused by incomplete movement during computation).
     • After the vector computation is complete, move data from the UB to the GM (to ensure that the result data is ready).
  2. Synchronization rules for cyclic data movement and computation scenarios
     • Start new data movement after the previous computation is complete. Do not trigger new data movement when the previous computation is not complete (to prevent the old data in the UB from being overwritten).
     • Start new computation after the previous data is moved out. Do not trigger a new computation task when the previous data is not completely moved out from the UB (to avoid overwriting conflicts in the target memory area).

  The synchronization control process is shown in the following figure.

  

  In the preceding figure, ID1, ID2, ID3, ID4, ID5, and ID6 represent event IDs. Each event ID corresponds to the movement status of a piece of stored data, ensuring the correctness and consistency of data operations.

  Note the following:

  • You are advised to obtain the event ID through the AllocEventID or FetchEventID API to ensure its validity.
  • The number of event IDs is limited. After using an event ID, call ReleaseEventID immediately to release the event ID, preventing event ID exhaustion and ensuring normal system operating.
  • SetFlag and WaitFlag must be used in pairs, and their parameters must be completely the same (including the template parameters and event ID). If they do not match, the computation of the current core may be abnormal, or the execution of the operator on the next core may be affected, causing a timeout.

    For example, SetFlag<HardEvent::S_MTE3>(1) and SetFlag<HardEvent::MTE3_MTE1>(1) set different event IDs because their template parameters are different. They set the same event ID only when their template parameters and event IDs are the same.

  • Do not set the same event ID consecutively. Otherwise, the event status may be disordered or not correctly processed.
  • You are advised not to manually insert TEventIDs. Do not manually insert TEventIDs 6 and 7 because they may be reserved by the system or used for special purposes.

• Inter-core synchronization

  When different cores operate the same global memory, data dependency issues such as read-after-write, write-after-read, and write-after-write may occur. To avoid such issues, inter-core synchronization control is required.

  The inter-core synchronization control has the following modes, as shown in the following figure.

  • Mode 0: synchronization control between AI Cores. In the AIC scenario, all AIC cores are synchronized. The subsequent instructions of CrossCoreWaitFlag are executed only when all AIC cores execute CrossCoreSetFlag. In the AIV scenario, all AIV cores are synchronized. The subsequent instructions of CrossCoreWaitFlag are executed only when all AIV cores execute CrossCoreSetFlag.
  • Mode 1: synchronization control between AIV cores in the AI Core. The subsequent instructions of CrossCoreWaitFlag are executed only when both AIV cores run CrossCoreSetFlag.
  • Mode 2: synchronization control between AIC and AIV cores in the AI Core (1:2). The subsequent instructions of CrossCoreWaitFlag on the two AIV cores are executed only after CrossCoreSetFlag is executed on the AIC core, and vice versa.
  • Mode 4: synchronization control between AIC and AIV cores in the AI Core (1:1). AIV0 and AIV1 can trigger AIC wait independently. For example, the subsequent instructions of CrossCoreWaitFlag on AIV0 are executed only after the AIC core executes CrossCoreSetFlag, and vice versa.

  

  For example, after the AIC core moves the L0C computation result to the GM, the AIV core needs to move the GM data to the UB. In this case, you can use the CrossCoreSetFlag and CrossCoreWaitFlag commands to ensure that the data is successfully moved from the L0C to the GM and then from the GM to the UB. The following figure shows the process.

  

  The CrossCoreSetFlag and CrossCoreWaitFlag APIs are used together. The flag ID (flagId, that is, ID1 in the above figure) for inter-core synchronization needs to be passed in. Each ID corresponds to a counter whose initial value is 0. After CrossCoreSetFlag is executed, the value of the counter corresponding to the ID is incremented by 1. If the counter value is 0, CrossCoreWaitFlag is not executed. If the counter value is greater than 0, the counter decreases by 1 and subsequent instructions start to be executed. The value of flagId ranges from 0 to 10.

  Note the following:

  • Used in pairs

    CrossCoreSetFlag and CrossCoreWaitFlag must be used in pairs. Otherwise, the operator may time out.

  • Data consistency

    The template parameters and flagId of CrossCoreSetFlag must be the same as those of CrossCoreWaitFlag. Otherwise, they are considered as different flag IDs. For example, CrossCoreSetFlag<0x0, PIPE_MTE3>(0x8) and CrossCoreSetFlag<0x2, PIPE_FIX>(0x8) set different flag IDs.

  • Avoiding consecutive settings

    Setting the same flag ID consecutively is not allowed, preventing the counter status from being disordered.

  • Use conflict with high-level APIs

    This API is used to control inter-core synchronization in the internal implementation of the Matmul high-level APIs. Therefore, you are advised not to use this API and the Matmul high-level APIs at the same time. Otherwise, flag IDs may conflict.

  • Counter restrictions

    The counter of the same flag ID can be set for a maximum of 15 times.

  • Default pipeline type

    The instruction pipeline type does not need to be explicitly set for CrossCoreWaitFlag. PIPE_S is used by default.