MmadBitMode

Applicability

Product	Supported (Prototype Without Bias Input)	Supported (Prototype with Bias Input)
Atlas 350 Accelerator Card	√	√
Atlas A3 training product / Atlas A3 inference product	x	x
Atlas A2 training product / Atlas A2 inference product	x	x
Atlas 200I/500 A2 inference product	x	x
Atlas inference product AI Core	x	x
Atlas inference product Vector Core	x	x
Atlas training product	x	x

Function Usage

Function 1: Performs the matrix multiplication and addition (C += A × B) operation. The matrices A, B, and C are data in A2, B2, and CO1, respectively.

The data formats of matrices A, B, and C are ZZ, ZN, and NZ, respectively. For details about the data formats, see Data Layout Formats.

In the figure below, each square represents a fractal matrix. The Z-shaped black line represents the data traversal order, which starts in the upper left corner and ends in the lower right corner.

Matrix A: The row-major order is used in each fractal matrix and between fractal matrices. This is called ZZ format. The fractal shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

Matrix B: The column-major order is used in each fractal matrix while the row-major order is used between fractal matrices. This is called NZ format. The fractal shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

Matrix C: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. This is called ZN format. The fractal shape is 16 × 16, and the size is 256 elements.

The following is a simple example. It is assumed that the size of a fractal matrix is 2 × 2 (which does not comply with an actual situation and is merely used as an example), and the sizes of matrices A, B, and C are all 4 × 4.

0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Ordering sequence of matrix A: 0, 1, 4, 5, 2, 3, 6, 7, 8, 9, 12, 13, 10, 11, 14, 15.

Ordering sequence of matrix B: 0, 4, 1, 5, 2, 6, 3, 7, 8, 12, 9, 13, 10, 14, 11, 15.

Ordering sequence of matrix C: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

The data formats of matrices A, B, and C are NZ, ZN, and NZ, respectively.

Matrix A: The row-major order is used in each fractal matrix, while the column-major order is used between fractal matrices. This is called ZN format. The shape is 16 × (32 bytes/sizeof(AType)), and the size is 512 bytes.

Matrix B: The column-major order is used in each fractal matrix while the row-major order is used between fractal matrices. This is called NZ format. The shape is (32 bytes/sizeof(BType)) × 16, and the size is 512 bytes.

0	1	2	3
4	5	6	7
8	9	10	11
12	13	14	15

Ordering sequence of matrix A: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

Ordering sequence of matrix B: 0, 4, 1, 5, 2, 6, 3, 7, 8, 12, 9, 13, 10, 14, 11, 15.

Ordering sequence of matrix C: 0, 1, 4, 5, 8, 9, 12, 13, 2, 3, 6, 7, 10, 11, 14, 15.

Function 2: For the Atlas 350 Accelerator Card, matrix multiplication with a scaling function is also supported. The formula is as follows: C = (ScaleA ⊗ A) × (ScaleB ⊗ B) + C. ScaleA and ScaleB are loaded through LoadData2DMX.

The fractal format of ScaleA is ZZ. The shape is (16, 2), and the data type is fp8_e8m0_t.
The fractal format of ScaleB is NN. The shape is (2, 16), and the data type is fp8_e8m0_t.

The following figure shows the fractal layout and resizing principles of ScaleA and ScaleB when the data types of matrices A and B are both fp4x2_e2m1_t.

Prototype

Without bias input

        
             template <typename T, typename U, typename S>
__aicore__ inline void Mmad(const LocalTensor<T>& dst, const LocalTensor<U>& fm, const LocalTensor<S>& filter, const MmadBitModeParams& mmadParams)

With bias input

        
             template <typename T, typename U, typename S, typename V>
__aicore__ inline void Mmad(const LocalTensor<T>& dst, const LocalTensor<U>& fm, const LocalTensor<S>& filter, const LocalTensor<V>& bias, const MmadBitModeParams& mmadParams)

Parameters

**Table 1** Template parameters
Parameter	Description
T	Data type of the destination operand.
U	Data type of the left matrix.
S	Data type of the right matrix.
V	Data type of the bias matrix.

**Table 2** Parameters
Parameter	Input/Output	Description
dst	Output	Destination operand. It represents the result matrix, which must be of type LocalTensor, and its TPosition is CO1. The start address of LocalTensor must be 256-element aligned.
fm	Input	Source operand. It represents the left matrix A, which must be of type LocalTensor, and its TPosition is A2. The start address of LocalTensor must be 512-byte aligned.
filter	Input	Source operand. It represents the right matrix B, which must be of type LocalTensor, and its TPosition is B2. The start address of LocalTensor must be 512-byte aligned.
bias	Input	Source operand. It represents the bias matrix, which must be of type LocalTensor, and its TPosition is C2 or CO1. The start address of LocalTensor must be 128-byte aligned.
mmadParams	Input	Matrix multiplication parameter. For details about the definition of this parameter, see ${INSTALL_DIR}/include/ascendc/basic_api/interface/kernel_struct_mm.h. Replace ${INSTALL_DIR} with the actual path for storing files after the CANN software is installed. For details about the parameters in MmadBitModeParams, see Table 3.

**Table 3** Parameters in **MmadBitModeParams**
Parameter	Description
config0	A parameter of type uint64_t. It forms a union together with a bit-field struct called MmadBitModeConfig0 (represented as config0BitMode). It is initialized to 0. You can use the class method GetConfig0() to retrieve its value.
config0BitMode	A bit-field struct type called MmadBitModeConfig0. It forms a union together with config0. For details about the parameter, see Table 4.

The design principles of the MmadBitModeParams class parameters are as follows:

A union is a special data structure that allows different data types to be stored in the same memory location. All members of a union share the same memory space, and its size is determined by the largest member. At any given time, only one member can be used.

A bit-field is a special class member that allows precise control over how many bits each variable occupies in memory. In a struct, members are laid out from top to bottom, corresponding to memory from low bits to high bits.

The MmadBitModeParams class uses both union and bit-field methods. Parameters are expressed using bit values, with the bit-field struct automatically handling the exact number of bits for each input. By leveraging the union, multiple parameters can be combined and passed together. Only one input parameter is needed to contain all required information, and the underlying API only needs to receive a single parameter. Additionally, when you need to modify the value of a specific bit within the parameter, you can do so with loops and bitwise operations, without having to re-pass the entire parameter. This reduces scalar computations and improves performance.

The MmadBitModeParams class can be directly initialized using an object of the MmadBitModeParams struct type.

      
           __aicore__ inline MmadBitModeParams(const MmadBitModeParams &mmadParams_);

You can also use the Set function of each parameter to modify the parameter value. Because the union is used, you can directly modify config0 bit by bit to modify the parameters.

**Table 4** Parameters in the **MmadBitModeConfig0** structure
Parameter	Description
m	Height of the left matrix. Value range: m ∈ [0, 4095]. Default value: 0. This parameter is the lowest-order field in the bit-field struct. It occupies 12 bits. You can set and obtain its value using the SetM() and GetM() functions of the MmadBitModeParams class, respectively.
k	Width of the right matrix. Value range: k ∈ [0, 4095]. Default value: 0. This parameter is the second-order field in the bit-field struct. It occupies 12 bits. You can set and obtain its value using the SetK() and GetK() functions of the MmadBitModeParams class, respectively.
n	Width of the left matrix and height of the right matrix. Value range: n ∈ [0, 4095]. Default value: 0. This parameter is the third-order field in the bit-field struct. It occupies 12 bits. You can set and obtain its value using the SetN() and GetN() functions of the MmadBitModeParams class, respectively.
unitFlag	Reserved parameter. This parameter is reserved for future functions. You can use the default value. This parameter is the fourth-order field in the bit-field struct. It occupies 2 bits. You can set and obtain its value using the SetUnitFlag() and GetUnitFlag() functions of the MmadBitModeParams class, respectively.
disableGemv	When M is 1, this parameter is used to control whether to enable General Matrix-Vector Multiplication (GEMV) for the Mmad computation. If the value is false, GEMV is enabled. If the value is true, GEMV is disabled. GEMV implements the multiplication of a matrix and a vector. After GEMV is enabled, the Mmad API reads data from the L0A buffer in ND format instead of ZZ format. This parameter is the fifth-order field in the bit-field struct. It occupies 1 bit. You can set and obtain its value using the SetDisableGemv() and GetDisableGemv() functions of the MmadBitModeParams class, respectively.
cmatrixSource	Controls whether the initial value of matrix C comes from C2 (hardware buffer for storing the bias). Default value: false. false: CO1 true: C2 Note: This parameter is invalid for the API with the bias input. Whether the initial value of matrix C comes from CO1 or C2 is determined based on the location of the bias input. This parameter is the sixth-order field in the bit-field struct. It occupies 1 bit. You can set and obtain its value using the SetCmatrixSource() and GetCmatrixSource() functions of the MmadBitModeParams class, respectively.
cmatrixInitVal	Controls whether the initial value of matrix C is 0. Default value: true. true: The initial value of matrix C is 0. false: The initial value of matrix C is specified by cmatrixSource. This parameter is the highest-order field in the bit-field struct. It occupies 1 bit. You can set and obtain its value using the SetCmatrixInitVal() and GetCmatrixInitVal() functions of the MmadBitModeParams class, respectively.

The parameters in the MmadBitModeConfig0 struct have the same meaning as the parameters with the same names in the MmadBitModeParams structure. For details, see Table 3.

**Table 5** Supported mixed precision of **dst**, fm, and **filter** (Atlas 350 Accelerator Card)
Left Matrix fm Type	Right Matrix filter Type	Result Matrix dst Type	Remarks
int8_t	int8_t	int32_t	Only matrix multiplication without resizing is supported.
half	half	float
float	float	float
bfloat16_t	bfloat16_t	float
fp8_e4m3fn_t	fp8_e4m3fn_t	float
fp8_e4m3fn_t	fp8_e5m2_t	float
fp8_e5m2_t	fp8_e4m3fn_t	float
fp8_e5m2_t	fp8_e5m2_t	float
hifloat8_t	hifloat8_t	float
fp4x2_e1m2_t	fp4x2_e1m2_t	float	Only matrix multiplication with resizing is supported.
fp4x2_e2m1_t	fp4x2_e1m2_t	float
fp4x2_e1m2_t	fp4x2_e2m1_t	float
fp4x2_e2m1_t	fp4x2_e2m1_t	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e4m3_t	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e5m2_t	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e4m3_t	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e5m2_t	float

**Table 6** Supported mixed precision of **dst**, fm, **filter**, and **bias** (Atlas 350 Accelerator Card)
Left Matrix fm Type	Right Matrix filter Type	bias Type	Result Matrix dst Type	Remarks
int8_t	int8_t	int32_t	int32_t	Only matrix multiplication without resizing is supported.
half	half	float	float
float	float	float	float
bfloat16_t	bfloat16_t	float	float
fp8_e4m3fn_t	fp8_e4m3fn_t	float	float
fp8_e4m3fn_t	fp8_e5m2_t	float	float
fp8_e5m2_t	fp8_e4m3fn_t	float	float
fp8_e5m2_t	fp8_e5m2_t	float	float
hifloat8_t	hifloat8_t	float	float
fp4x2_e1m2_t	fp4x2_e1m2_t	float	float	Only matrix multiplication with resizing is supported.
fp4x2_e2m1_t	fp4x2_e1m2_t	float	float
fp4x2_e1m2_t	fp4x2_e2m1_t	float	float
fp4x2_e2m1_t	fp4x2_e2m1_t	float	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e4m3_t	float	float
AscendC::mx_fp8_e4m3_t	AscendC::mx_fp8_e5m2_t	float	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e4m3_t	float	float
AscendC::mx_fp8_e5m2_t	AscendC::mx_fp8_e5m2_t	float	float

Restrictions

dst can only be located in CO1, fm in A2, and filter in B2.
If any of M, K, and N is 0, the instruction is not executed.
When M is 1, the GEMV function is enabled by default. In this case, the Mmad API reads data from L0A Buffer in ND format instead of ZZ format. Therefore, the left matrix needs to be directly arranged in ND format. For the Atlas 350 Accelerator Card, you can disable this function by setting disableGemv of MmadBitModeParams to true.
For details about the operand address alignment requirements, see General Address Alignment Restrictions.
The following uses an example to describe the arrangement of invalid and valid data.
When the data type is half and the dimensions are M = 30, K = 70, N = 40, there are 2 × 5 matrices with the size of 16 × 16 in A2, 5 × 3 matrices with the size of 16 × 16 in B2, and 2 × 3 matrices with the size of 16 × 16 in CO1. In this scenario, M, K, and N are not multiples of 16. In A2, the bottom-right matrix only has 14 × 6 valid elements, but it still occupies the full 16 × 16 space. The remaining positions are filled with invalid data, which is ignored during computation. In a 16 × 16 fractal data block, the arrangements of invalid and valid data are as follows.

Examples

For details about the example without matrix multiplication bias, see Mmad sample.

For details about the example with matrix multiplication bias, see sample of Mmad with matrix multiplication bias.

Parent topic: Cube Computation