Matmul Dequantization and Bias Adding

Applicable Products

Hardware Model	Supported or Not	Remarks
Atlas 350 accelerator card	x	-
Atlas A3 inference products/Atlas A3 training products	√	-
Atlas A2 training products/Atlas A2 inference products	√	-
Atlas training products	√	transposeA cannot be set to true. transposeB cannot be set to false. hasBias cannot be set to false. outDataType cannot be ACL_BF16.
Atlas inference products	√	transposeA cannot be set to true. transposeB cannot be set to false. hasBias cannot be set to false. outDataType cannot be ACL_BF16.
Atlas 200I/500 A2 inference products	√	The input and output tensors of the bf16 data type are not supported.

Description

Matmul and dequantization. The hasBias parameter is used to control whether to add bias.

Formula

Inputs tensors A (x) and B (weight), dequantization step deqScale, output tensor C, dequantization processing function dequant:

$\text{[math]}$

Bias matrix is added:

$\text{[math]}$

Parameter Configuration

Member	Value Range	Remarks
transposeA	false/true	If this parameter is set to true, some scenarios are not supported. For details, see Specifications.
transposeB	false/true	Varies depending on hardware.
hasBias	false/true	Varies depending on hardware.
outDataType	ACL_FLOAT16/ACL_BF16	Varies depending on hardware. Data type of the output tensor
enAccum	false	-
matmulType	MATMUL_UNDEFINED	-
quantMode	QUANT_UNDEFINED/PER_CHANNEL	-

Input

Parameter	Dimension	Data Type	Format	Description
x	[m, k]/[batch, m, k]	int8	ND	Matrix A for matrix multiplication.
weight	[k, n]/[batch, k, n]/[1, n / 16, k, 16]/[batch, n / 16, k, 16]	int8	ND/NZ	Weight of matrix B for matrix multiplication.
bias	[1, n]/[n]/[batch, n]	int32	ND	Added bias matrix Valid when hasBias is set to true.
deqScale	[1, n]/[n]/[batch, n]	int64/uint64/float	ND	Dequantization step The actually occupied memory must be 32-byte aligned.

Output

Parameter	Dimension	Data Type	Format	Description
output	[m, n]/[batch, m, n]	float16/bf16	ND	Matrix multiplication and dequantization result

Description

The following table lists the combinations of input and output attributes. The combinations that are not listed in the table are not supported.

Figure 1 Combinations of input and output attributes

Atlas inference products: Combinations 7 to 15 are not supported.
Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products: Combinations 10 to 15 are not supported.
When transposeA is set to true, combinations 2, 5, 8, 11, and 14 are not supported.

When the weight has four dimensions, if transposeB is false, n is an integer multiple of 32, and k is an integer multiple of 16; if transposeB is true, k is an integer multiple of 32, and n is an integer multiple of 16.
Atlas training products: Combinations 1 to 9 are not supported.

OP Usage and Typical Scenarios

For details about how to use OP, see the usage process in Operator Usage Guide (ATB C++ APIs). For details about how to construct the Operation parameter in Single-operator, see the following parameter construction part.

// Parameter construction
atb::infer::LinearParam param;
param.transposeA = false;
param.transposeB = false;
param.hasBias = true;
param.outDataType = ACL_FLOAT16;
param.enAccum = false;
param.matmulType = MATMUL_UNDEFINED;
param.quantMode = PER_CHANNEL;

# Example
>>> x
tensor([[1, 2],
        [3, 4]])
>>> weight
tensor([[1, 2, 3],
        [4, 5, 6]])
>>> bias
tensor([1, 2, 3])
>>> deqScale
tensor([1, 2, 3])
>>> output
tensor([[10, 28, 54],
        [20, 56, 108]])
# 10 = (1 * 1 + 2 * 4 + 1) * 1
# 28 = (1 * 2 + 2 * 5 + 2) * 2
# 54 = (1 * 3 + 2 * 6 + 3) * 3
# 20 = (3 * 1 + 4 * 4 + 1) * 1
# 56 = (3 * 2 + 4 * 5 + 2) * 2
# 108 = (3 * 3 + 4 * 6 + 3) * 3

Parent topic: Functions