Fusion of Matmul and Add

Applicable Products

Hardware Model	Supported	Remarks
Atlas 350 accelerator cards	x	-
Atlas A3 inference products/Atlas A3 training products	√	-
Atlas A2 training products/Atlas A2 inference products	√	-
Atlas training products	x	-
Atlas inference products	x	-
Atlas 200I/500 A2 inference products	√	The input and output tensors of the bf16 data type are not supported.

Description

Fuses matmul and add. This function is similar to the bias addition in function 1 but performs better.

Formula

Multiplies tensors A (x) and B (weight), adds the bias matrix, and outputs tensor C.

$\text{[math]}$

Parameters

Member	Value Range	Remarks
transposeA	false/true	If this parameter is set to true, some scenarios are not supported. For details, see Specifications.
transposeB	false/true	-
hasBias	true	-
outDataType	ACL_DT_UNDEFINED	-
enAccum	false	-
matmulType	MATMUL_UNDEFINED	-
quantMode	QUANT_UNDEFINED	-

Input

Parameter	Dimension	Data Type	Format	Description
x	[m, k]/[batch, m, k]	float16/bf16	ND	Matrix A for matrix multiplication.
weight	[k, n]/[batch, k, n]	float16/bf16	ND	Weight of matrix B for matrix multiplication.
bias	[1, n]/[n]/[batch, n]	float	ND	Added bias matrix.

Output

Parameter	Dimension	Data Type	Format	Description
output	[m, n]/[batch, m, n]	float16/bf16	ND	Matrix multiplication result.

Specifications

Figure 1 Combinations of input and output attributes

If transposeA is set to true, combinations 2 and 5 are not supported.

OP Usage and Typical Scenarios

For details about how to use OP, see the usage process in Operator Usage Guide (ATB C++ APIs). For details about how to construct the Operation parameter in Single Operator, see the following parameter construction part.

// Parameter construction
atb::infer::LinearParam param;
param.transposeA = false;
param.transposeB = false;
param.hasBias = true;
param.outDataType = ACL_DT_UNDEFINED;
param.enAccum = false;
param.matmulType = MATMUL_UNDEFINED;
param.quantMode = QUANT_UNDEFINED;

# Example
>>> x
tensor([[1, 2],
        [3, 4]])
>>> weight
tensor([[1, 2, 3],
        [4, 5, 6]])
>>> bias
tensor([1, 2, 3])
>>> output
tensor([[10, 14, 18],
        [20, 28, 36]])
# 10 = 1 * 1 + 2 * 4 + 1
# 14 = 1 * 2 + 2 * 5 + 2
# 18 = 1 * 3 + 2 * 6 + 3
# 20 = 3 * 1 + 4 * 4 + 1
# 28 = 3 * 2 + 4 * 5 + 2
# 36 = 3 * 3 + 4 * 6 + 3

Function Constraints

Some scenarios are not supported when transposeA is set to true.
hasBias = true.
outDataType = ACL_DT_UNDEFINED.
enAccum = false.

Parent topic: Functions