LinearSparseOperation

Description

Sparse quantization linear.

The implementation function of this operator is similar to that of quantization linear. The difference is that the sparse quantization operator uses a compression tool to compress the input weight in advance to improve operator performance. The tilingK and tilingN parameters are determined by the compression algorithm. Currently, only the value 8 is supported. Currently, this operator can be used only in the Atlas inference products.

Definition

struct LinearSparseParam{
    bool transposeA = false;
    bool transposeB = true;
    uint32_t tilingK = 1;
    uint32_t tilingN = 1;
    uint8_t rsv[12] = {0};
};

Parameters

Member	Type	Default Value	Description
transposeA	bool	false	Whether to transpose matrix A. In quantization scenarios, only false is supported for products except for the Atlas A2 inference products.
transposeB	bool	true	Whether to transpose matrix B. In dequantization scenarios, only true is supported for products except for the Atlas A2 inference products.
tilingK	uint32_t	1	Compression parameter, which is determined by the external compression algorithm. Currently, the value can only be 8.
tilingN	uint32_t	1	Compression parameter, which is determined by the external compression algorithm. Currently, the value can only be 8.
rsv[12]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[m, k]	int8	ND	Matrix A for matrix multiplication. m must be less than or equal to 256, and k must be an integer multiple of 64 and greater than 256.
weight	[c]	int8	ND/NZ	Weight, matrix B for matrix multiplication. Weight compressed by the compression tool. The value of shape size c is greater than 0 and less than or equal to k × n.
bias	[1, n] or [n]	int32	ND	Added bias matrix n is an integer multiple of 64 and is greater than or equal to 128.
deqScale	[1, n] or [n]	uint64/int64	ND	Scale of dequantization. This parameter is input during quantization.
compressIdx	[x]	int8	ND	Compression index generated during weight compression. x is calculated as follows: x = k_index * n_index * 8 k_index = ceil(k1 / tilingK) n_index = ceil(n1 / tilingN) k1 = k / 32 n1 = n / 16 The ceil() function rounds up the final value.

Output

Parameter	Dimension	Data Type	Format	Description
output	[m, n]	float16	ND	Output tensor. The number of dimensions is the same as that of x.

Restrictions

Use transposeA and transposeB to configure the original x/weight dimensions to meet the dimension relationship of matrix multiplication.
The dimension of the input weight (c)is calculated by a compression tool (such as ModelSlim), and is greater than 0 but less than k × n.
This operator is supported only by the Atlas inference products.
The tilingK and tilingN parameters are determined by the compression algorithm. Currently, only the value 8 is supported.

Parent topic: atb/infer_op_params.h