The storage units on the AI Core are used to store the source operands and destination operands for vector computation and matrix computation. The alignment requirements for each type of storage unit are shown in . Therefore, the starting address alignment requirements for C API operands must be consistent with the alignment requirements of these storage units. Note that if the operand starting address alignment requirement is explicitly specified in an interface, the specification in the specific API takes precedence.
Table 1 Alignment Requirements for Different Storage Units
When using high-dimensional tiling computation interfaces, to save address space, developers can allocate a block of memory for both the source operand and the destination operand to use simultaneously (that is, address overlap). The following constraints must be observed:
- Within a single iteration: The source operand and the destination operand must be 100% fully overlapped. Partial overlap is not supported.
- Across multiple iterations: The destination operand of a previous iteration overlapping with the source operand of a subsequent iteration is not supported. For example, the destination operand of the Nth iteration is the source operand of the (N+1)th iteration (as shown in the figure below). In this case, the Nth iteration may overwrite the value of the source operand, causing unexpected results. Specifically, for some binary computation APIs (asc_add, asc_sub, asc_mul, and so on), when the data type is half, int32_t, or float, the destination operand of a previous iteration overlapping with the source operand of a subsequent iteration is supported: this applies only to the case where the destination operand overlaps with the second source operand, and src1RepStride or dstRepStride must be 0.
Figure 1 Address Overlap Example (Not Supported)
[object Object]
[object Object]
Using high-dimensional tiling computation APIs fully leverages hardware advantages, allowing developers to control the iterative execution of instructions and the address intervals of operands, making the functionality more flexible.
Vector computation is completed by the Vector computation unit. The source operands and destination operands of vector computation are both stored through the Unified Buffer (UB). In each iteration, the Vector computation unit fetches 8 DataBlocks from the UB (each DataBlock has internally contiguous addresses and a length of 32 bytes), performs computation, and writes the results to the corresponding 8 DataBlocks. The following figure shows a schematic diagram of Exp computation on 8 DataBlocks within a single iteration.
Figure 1 Schematic Diagram of Exp Computation on 8 DataBlocks Within a Single Iteration
- The vector computation APIs allow developers to configure the repeat count through repeatTime, thereby controlling the multiple iterative execution of instructions. Assuming repeatTime is set to 2, the Vector computation unit performs 2 iterations of computation, producing 2 * 8 (8 DataBlocks per iteration) * 32 bytes (32 bytes per DataBlock) = 512 bytes of results. If the data type is half, 256 elements are computed. The following figure shows a schematic diagram of 2 iterations of Exp computation. Due to hardware limitations, repeatTime cannot exceed 255.[object Object]
Figure 2 2 Iterations of Exp Computation
- For data within the same iteration, the mask parameter can be used to perform mask operations to control the actual number of elements participating in the computation. The following figure shows a schematic diagram of controlling which elements participate in the computation bit by bit through the mask in bit-by-bit mode during Abs computation. A value of 1 indicates participation in the computation, and 0 indicates non-participation.[object Object]
Figure 3 Schematic Diagram of Mask Operation Through the Mask Parameter (Using the float Data Type as an Example)
- The Vector computation unit also supports interval-based vector computation, which is configured through dataBlockStride (the address stride between different DataBlocks within a single iteration) and repeatStride (the address stride of the same DataBlock between adjacent iterations).
dataBlockStride [object Object] If you need to control the stride of data processing within a single iteration, you can set the address stride dataBlockStride between different DataBlocks within the same iteration. The following figure shows a schematic diagram of a non-contiguous scenario within a single iteration. In this example, the dataBlockStride of the source operand is set to 2, indicating that the address stride (interval between starting addresses) between different DataBlocks within a single iteration is 2 DataBlocks.[object Object] Figure 4 Schematic Diagram of Non-contiguous Scenario Within a Single Iteration
repeatStride
When repeatTime is greater than 1 and multiple iterations are required to complete the vector computation, you can reasonably set the value of repeatStride (the address stride of the same DataBlock between adjacent iterations) based on different usage scenarios.[object Object]
The following figure shows a schematic diagram of a non-contiguous scenario across multiple iterations. In this example, the repeatStride of both the source operand and the destination operand is set to 9, indicating that the interval between the starting addresses of the same DataBlock across adjacent iterations is 9 DataBlocks. The same DataBlock refers to a DataBlock at the same position within the iteration. For example, src1 and src9 in the following figure are in adjacent iterations, both at the first DataBlock position within the iteration, and the interval between them is the value of repeatStride. [object Object]
Figure 5 Schematic Diagram of Non-contiguous Scenario Across Multiple Iterations
The following sections provide detailed configuration instructions and examples for dataBlockStride, repeatStride, and mask.
dataBlockStride refers to the address stride between different DataBlocks within the same iteration.
- For contiguous computation, dataBlockStride is set to 1, and the 8 DataBlocks within the same iteration are processed contiguously.
- For non-contiguous computation, dataBlockStride is greater than 1 (for example, 2). When reading data, there is a one-DataBlock gap between different DataBlocks within the same iteration, as shown in the following figure. [object Object]
Figure 6 Examples of Different dataBlockStride Values
repeatStride refers to the address stride of the same DataBlock between adjacent iterations.
Contiguous computation scenario: Assume that a Tensor is defined for both the destination operand and the source operand to use simultaneously (that is, address overlap), and repeatStride is set to 8. In this case, the Vector computation unit reads 8 contiguous DataBlocks in the first iteration, and reads the next 8 contiguous DataBlocks in the second iteration. All input data can be computed through multiple iterations.
Non-contiguous computation scenario: When repeatStride is greater than 8 (for example, 10), the data read by the Vector computation unit between adjacent iterations is not contiguous in address, resulting in a gap of 2 DataBlocks.
Repeated computation scenario: When repeatStride is set to 0, the Vector computation unit repeatedly reads and computes the first 8 contiguous DataBlocks.
Partial repeated computation: When repeatStride is greater than 0 and less than 8, some data between adjacent iterations is repeatedly read and computed by the Vector computation unit. This scenario is generally not involved in common use cases.
mask is used to control the elements that participate in the computation within each iteration. You can control which elements participate in the computation bit by bit. A bit value of 1 indicates participation in the computation, and 0 indicates non-participation.
The actual value range of mask is related to the data type of the operand. When the operand is 16-bit, mask contains 2 uint64_t values, with mask0 and mask1 in [0, 2^64-1] and not both 0 at the same time; when the operand is 32-bit, mask contains 1 uint64_t value, with mask0 in (0, 2^64-1]; when the operand is 64-bit, mask contains 1 uint64_t value, with mask0 in (0, 2^32-1]. [object Object]
The following is a specific sample:
The result example is as follows:
The mask process is as follows:[object Object]
mask={6148914691236517205, 6148914691236517205} (Note: 6148914691236517205 represents the 64-bit binary number 0b010101....01. The mask is arranged from the low bit to the high bit.)
The result example is as follows:
The mask process is as follows:[object Object]
mask={6148914691236517205, 0} (Note: 6148914691236517205 represents the 64-bit binary number 0b010101....01)
The compiler supports allocating memory in array mode. However, the following constraints must be observed:
- Currently, only the Atlas A3 training product/Atlas A3 inference product and the Atlas A2 training product/Atlas A2 inference product are supported.
- The array allocation method and the asc_get_phy_buf_addr API interface cannot be used together. Otherwise, address overlap may occur.
- Multi-dimensional arrays and nested arrays are not supported.
- When encapsulated in a data structure, implicit construction is not supported.
- Dynamic arrays are not supported.
The basic usage is as follows: