Fusion of Matmul and InplaceAdd

Applicable Products

Hardware Model	Supported or Not
Atlas 350 accelerator card	x
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Fuses matmul with inplaceAdd to add the matrix multiplication result to the accumulation matrix. Compared with the operator concatenation solution, this solution provides better performance.

Formula

Input tensors A (x) and B (weight) to output the output tensor., accumulated tensor C (accum).

$\text{[math]}$

Parameter Configuration

Member	Value Range	Remarks
transposeA	false/true	If this parameter is set to true, some scenarios are not supported. For details, see Specifications. If the input data type is float16, transposeA cannot be set to true.
transposeB	false/true	-
hasBias	false	-
outDataType	ACL_DT_UNDEFINED	-
enAccum	true	-
matmulType	MATMUL_UNDEFINED	-
quantMode	QUANT_UNDEFINED	-

Input

Parameter	Dimension	Data Type	Format	Description
x	[m, k]/[batch, m, k]	float16/bf16	ND	Matrix A for matrix multiplication.
weight	[k, n]/[batch, k, n]	float16/bf16	ND	Weight of matrix B for matrix multiplication.
accum	[m, n]/[batch, m, n]	float	ND	Added matrix

Output

Parameter	Dimension	Data Type	Format	Description
output	[m, n]/[batch, m, n]	float	ND	Accumulation matrix, which is the same tensor as accum. That is, the calculation result is written in place.

Description

The following table lists the combinations of input and output attributes. The combinations that are not listed in the table are not supported.

Figure 1 Combinations of input and output attributes

If transposeA is set to true, combinations 2, 5, 8, and 11 are not supported.

OP Usage and Typical Scenarios

For details about how to use OP, see the usage process in Operator Usage Guide (ATB C++ APIs). For details about how to construct the Operation parameter in Single-operator, see the following parameter construction part.

// Parameter construction
atb::infer::LinearParam param;
param.transposeA = false;
param.transposeB = false;
param.hasBias = false;
param.outDataType = ACL_DT_UNDEFINED;
param.enAccum = true;
param.matmulType = MATMUL_UNDEFINED;
param.quantMode = QUANT_UNDEFINED;

# Example
>>> x
tensor([[1, 2],
        [3, 4]])
>>> weight
tensor([[1, 2, 3],
        [4, 5, 6]])
>>> accum
tensor([[1, 2, 3],
        [4, 5, 6])
>>> output
tensor([[10, 14, 18],
        [23, 31, 39]])
# 10 = 1 * 1 + 2 * 4 + 1
# 14 = 1 * 2 + 2 * 5 + 2
# 18 = 1 * 3 + 2 * 6 + 3
# 23 = 3 * 1 + 4 * 4 + 4
# 31 = 3 * 2 + 4 * 5 + 5
# 39 = 3 * 3 + 4 * 6 + 6

Function Constraints

Some scenarios are not supported when transposeA is set to true.
hasBias = false.
outDataType = ACL_DT_UNDEFINED.
enAccum = true.

Parent topic: Functions