Mmad

Applicability

Product	Supported (Prototype Without Bias Input)	Supported (Prototype with Bias Input)
Atlas 350 Accelerator Card	√	√
Atlas A3 training product / Atlas A3 inference product	√	√
Atlas A2 training product / Atlas A2 inference product	√	√
Atlas 200I/500 A2 inference product	√	√
Atlas inference product AI Core	√	x
Atlas inference product Vector Core	x	x
Atlas training product	√	x

Function Usage

Function 1: Performs the matrix multiplication and addition (C += A × B) operation. The matrices A, B, and C are data in A2, B2, and CO1, respectively.

The data formats of matrices A, B, and C are ZZ, ZN, and NZ, respectively. For details about the data formats, see Data Layout Formats.

In the figure below, each square represents a fractal matrix. The Z-shaped black line represents the data traversal order, which starts in the upper left corner and ends in the lower right corner.

Matrix A: The row-major order is used in each fractal matrix and between fractal matrices. This is called ZZ format. The fractal shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

Matrix B: The column-major order is used in each fractal matrix while the row-major order is used between fractal matrices. This is called NZ format. The fractal shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

Matrix C: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. This is called ZN format. The fractal shape is 16 × 16, and the size is 256 elements.

The following is a simple example. It is assumed that the size of a fractal matrix is 2 × 2 (which does not comply with an actual situation and is merely used as an example), and the sizes of matrices A, B, and C are all 4 × 4.

0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Ordering sequence of matrix A: 0, 1, 4, 5, 2, 3, 6, 7, 8, 9, 12, 13, 10, 11, 14, 15.

Ordering sequence of matrix B: 0, 4, 1, 5, 2, 6, 3, 7, 8, 12, 9, 13, 10, 14, 11, 15.

Ordering sequence of matrix C: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

The data formats of matrices A, B, and C are NZ, ZN, and NZ, respectively.

Matrix A: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. This is called ZN format. The shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

Matrix B: The column-major order is used in each fractal matrix while the row-major order is used between fractal matrices. This is called NZ format. The shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Ordering sequence of matrix A: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

Ordering sequence of matrix B: 0, 4, 1, 5, 2, 6, 3, 7, 8, 12, 9, 13, 10, 14, 11, 15.

Ordering sequence of matrix C: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

Function 2: For the Atlas 350 Accelerator Card, matrix multiplication with a scaling function is also supported. The formula is as follows: C = (ScaleA ⊗ A) × (ScaleB ⊗ B) + C. ScaleA and ScaleB are loaded through LoadData2DMX.

The fractal format of ScaleA is ZZ. The shape is (16, 2), and the data type is fp8_e8m0_t.
The fractal format of ScaleB is NN. The shape is (2, 16), and the data type is fp8_e8m0_t.

The following figure shows the fractal layout and resizing principles of ScaleA and ScaleB when the data types of matrices A and B are both fp4x2_e2m1_t.

Prototype

Without bias input

        
             template <typename T, typename U, typename S>
__aicore__ inline void Mmad(const LocalTensor<T>& dst, const LocalTensor<U>& fm, const LocalTensor<S>& filter, const MmadParams& mmadParams)

With bias input

        
             template <typename T, typename U, typename S, typename V>
__aicore__ inline void Mmad(const LocalTensor<T>& dst, const LocalTensor<U>& fm, const LocalTensor<S>& filter, const LocalTensor<V>& bias, const MmadParams& mmadParams)

Parameters

**Table 1** Template parameters
Parameter	Description
T	Data type of the destination operand.
U	Data type of the left matrix.
S	Data type of the right matrix.
V	Data type of the bias matrix.

**Table 2** Parameters
Parameter	Input/Output	Description
dst	Output	Destination operand. It represents the result matrix, which must be of type LocalTensor, and its TPosition is CO1. The start address of LocalTensor must be 256-element-aligned.
fm	Input	Source operand. It represents the left matrix A, which must be of type LocalTensor, and its TPosition is A2. The start address of LocalTensor must be 512-byte aligned.
filter	Input	Source operand. It represents the right matrix B, which must be of type LocalTensor, and its TPosition is B2. The start address of LocalTensor must be 512-byte aligned.
bias	Input	Source operand. It represents the bias matrix, which must be of type LocalTensor, and its TPosition is C2 or CO1. The start address of LocalTensor must be 128-byte aligned.
mmadParams	Input	Matrix multiplication parameter. For details about the definition of this parameter, see ${INSTALL_DIR}/include/ascendc/basic_api/interface/kernel_struct_mm.h. Replace ${INSTALL_DIR} with the actual path for storing files after the CANN software is installed. For details about the parameters in MmadParams, see Table 3.

**Table 3** Parameters in the **MmadParams** structure
Parameter	Description
m	Height of the left matrix. Value range: m ∈ [0, 4095]. Default value: 0.
n	Width of the right matrix. Value range: n ∈ [0, 4095]. Default value: 0.
k	Width of the left matrix and height of the right matrix. Value range: k ∈ [0, 4095]. Default value: 0.
cmatrixInitVal	Controls whether the initial value of matrix C is 0. Default value: true. true: The initial value of matrix C is 0. false: The initial value of matrix C is specified by cmatrixSource.
cmatrixSource	Controls whether the initial value of matrix C comes from C2 (hardware buffer for storing the bias). Default value: false. false: CO1 true: C2 For the Atlas training product , this parameter can only be set to false. For the Atlas inference product AI Core, this parameter can only be set to false. For the Atlas A2 training product / Atlas A2 inference product , this parameter can be true or false. For the Atlas A3 training product / Atlas A3 inference product , this parameter can be true or false. For the Atlas 200I/500 A2 inference product , this parameter can be true or false. For the Atlas 350 Accelerator Card, this parameter can be true or false. Note: This parameter is invalid for the API with the bias input. Whether the initial value of matrix C comes from CO1 or C2 is determined based on the location of the bias input.
isBias	This parameter is deprecated. Do not use this parameter in new development. To add up the initial matrices, use the API with bias. You can also use the cmatrixInitVal and cmatrixSource parameters to configure the initial value source of matrix C. You are advised to use the API with bias, which is easier to configure than cmatrixInitVal and cmatrixSource. Controls whether the initial matrix should be accumulated. The default value is false. The options are as follows: false: matrix multiplication. The initial matrix is not accumulated. Its formula is C = A × B. true: matrix multiply-accumulate. The initial matrix is accumulated. Its formula is C += A × B.
disableGemv	When M is 1, this parameter is used to control whether to enable General Matrix-Vector Multiplication (GEMV) for the Mmad computation. If the value is false, GEMV is enabled. If the value is true, GEMV is disabled. GEMV implements the multiplication of a matrix and a vector. After GEMV is enabled, the Mmad API reads data from the L0A buffer in ND format instead of ZZ format. This parameter is supported only by the following models: Atlas 350 Accelerator Card
unitFlag	Fine-grained parallelism between Mmad and Fixpipe instructions. After this function is enabled, the computation result is moved out each time the hardware completes a fractal computation. This function is not applicable to scenarios where accumulation is performed in the L0C buffer. The options are as follows: 0: Value reserved. 2: The unitFlag function is enabled. After the hardware executes the instruction, the unitFlag function is not disabled. 3: The unitFlag function is enabled. After the hardware executes the instruction, the unitFlag function is disabled. When this function is enabled, set the unitFlag of the Mmad instruction to 3 for the last fractal and to 2 for other fractals. This parameter is supported only by the following models: Atlas 350 Accelerator Card Atlas A2 training product / Atlas A2 inference product Atlas A3 training product / Atlas A3 inference product
fmOffset	Reserved parameter. This parameter is reserved for future functions. You can use the default value.
enSsparse
enWinogradA
enWinogradB
kDirectionAlign	Controls whether alignment is required. The default value is false. For the Atlas training product , this parameter can only be set to false. For the Atlas inference product AI Core, this parameter can only be set to false. For the Atlas A2 training product / Atlas A2 inference product , this parameter can only be set to false. For the Atlas A3 training product / Atlas A3 inference product , this parameter can only be set to false. For the Atlas 200I/500 A2 inference product , this parameter can only be set to false. For the Atlas 350 Accelerator Card, this parameter can only be set to false.

**Table 4** Supported mixed precision of **dst**, fm, and **filter** ( Atlas training product )
Left Matrix fm Type	Right Matrix filter Type	Result Matrix dst Type
uint8_t	uint8_t	uint32_t
int8_t	int8_t	int32_t
uint8_t	int8_t	int32_t
half	half	half NOTE: The mixed precision of this type cannot reach double 1‰, and later processor versions do not support this type conversion. You are advised to use half input and float output. The double 1‰ means that the error between each actual data record and the true value does not exceed 1‰, and the total number of data records whose error exceeds 1‰ does not exceed 1‰ of the total number of data records.
half	half	float

**Table 5** Supported mixed precision of **dst**, fm, and **filter** ( Atlas inference product AI Core)
Left Matrix fm Type	Right Matrix filter Type	Result Matrix dst Type
int8_t	int8_t	int32_t
uint8_t	int8_t	int32_t
uint8_t	uint8_t	int32_t
half	half	half NOTE: The mixed precision of this type cannot reach double 1‰, and later processor versions do not support this type conversion. You are advised to use half input and float output. The double 1‰ means that the error between each actual data record and the true value does not exceed 1‰, and the total number of data records whose error exceeds 1‰ does not exceed 1‰ of the total number of data records.
half	half	float
int4b_t	int4b_t	int32_t

**Table 6** Supported mixed precision of **dst**, fm, and **filter** ( Atlas 200I/500 A2 inference product )( Atlas A2 training product / Atlas A2 inference product )( Atlas A3 training product / Atlas A3 inference product )
Left Matrix fm Type	Right Matrix filter Type	Result Matrix dst Type
int8_t	int8_t	int32_t
half	half	float
float	float	float
bfloat16_t	bfloat16_t	float
int4b_t	int4b_t	int32_t

**Table 7** Supported mixed precision of **dst**, fm, and **filter** (Atlas 350 Accelerator Card)
Left Matrix fm Type	Right Matrix filter Type	Result Matrix dst Type	Remarks
int8_t	int8_t	int32_t	Only matrix multiplication without resizing is supported.
half	half	float
float	float	float
bfloat16_t	bfloat16_t	float
fp8_e4m3fn_t	fp8_e4m3fn_t	float
fp8_e4m3fn_t	fp8_e5m2_t	float
fp8_e5m2_t	fp8_e4m3fn_t	float
fp8_e5m2_t	fp8_e5m2_t	float
hifloat8_t	hifloat8_t	float
fp4x2_e1m2_t	fp4x2_e1m2_t	float	Only matrix multiplication with resizing is supported.
fp4x2_e2m1_t	fp4x2_e1m2_t	float
fp4x2_e1m2_t	fp4x2_e2m1_t	float
fp4x2_e2m1_t	fp4x2_e2m1_t	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e4m3_t	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e5m2_t	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e4m3_t	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e5m2_t	float

**Table 8** Supported mixed precision of **dst**, fm, **filter**, and **bias** ( Atlas 200I/500 A2 inference product )( Atlas A2 training product / Atlas A2 inference product )( Atlas A3 training product / Atlas A3 inference product )
Left Matrix fm Type	Right Matrix filter Type	bias Type	Result Matrix dst Type
int8_t	int8_t	int32_t	int32_t
half	half	float	float
float	float	float	float
bfloat16_t	bfloat16_t	float	float

**Table 9** Supported mixed precision of **dst**, fm, **filter**, and **bias** (Atlas 350 Accelerator Card)
Left Matrix fm Type	Right Matrix filter Type	bias Type	Result Matrix dst Type	Remarks
int8_t	int8_t	int32_t	int32_t	Only matrix multiplication without resizing is supported.
half	half	float	float
float	float	float	float
bfloat16_t	bfloat16_t	float	float
fp8_e4m3fn_t	fp8_e4m3fn_t	float	float
fp8_e4m3fn_t	fp8_e5m2_t	float	float
fp8_e5m2_t	fp8_e4m3fn_t	float	float
fp8_e5m2_t	fp8_e5m2_t	float	float
hifloat8_t	hifloat8_t	float	float
fp4x2_e1m2_t	fp4x2_e1m2_t	float	float	Only matrix multiplication with resizing is supported.
fp4x2_e2m1_t	fp4x2_e1m2_t	float	float
fp4x2_e1m2_t	fp4x2_e2m1_t	float	float
fp4x2_e2m1_t	fp4x2_e2m1_t	float	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e4m3_t	float	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e5m2_t	float	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e4m3_t	float	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e5m2_t	float	float

Restrictions

dst can only be located in CO1, fm in A2, and filter in B2.
If any of M, K, and N is 0, the instruction is not executed.
When M is 1, the GEMV function is enabled by default. In this case, the Mmad API reads data from L0A Buffer in ND format instead of ZZ format. Therefore, the left matrix needs to be directly arranged in ND format. For the Atlas 350 Accelerator Card, you can disable this function by setting disableGemv of MmadParams to true.
For details about the operand address alignment requirements, see General Address Alignment Restrictions.
The following uses an example to describe the arrangement of invalid and valid data.
When the data type is half and the dimensions are M = 30, K = 70, N = 40, there are 2 × 5 matrices with the size of 16 × 16 in A2, 5 × 3 matrices with the size of 16 × 16 in B2, and 2 × 3 matrices with the size of 16 × 16 in CO1. In this scenario, M, K, and N are not multiples of 16. In A2, the bottom-right matrix only has 14 × 6 valid elements, but it still occupies the full 16 × 16 space. The remaining positions are filled with invalid data, which is ignored during computation. In a 16 × 16 fractal data block, the arrangements of invalid and valid data are as follows.

Examples

For details about the example without matrix multiplication bias, see the Mmad sample.

For details about the example with matrix multiplication bias, see the sample of Mmad with matrix multiplication bias.

Parent topic: Cube Computation