LinearSparseOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	x
Atlas A3 inference products/Atlas A3 training products	x
Atlas A2 training products/Atlas A2 inference products	x
Atlas training products	x
Atlas inference products	√
Atlas 200I/500 A2 inference products	x

Description

Implements a linear layer that runs on sparse, quantized data.

This operator functions the same as the linear quantization. The difference is that the sparse quantization operator uses a compression tool to compress the input weight in advance to improve operator performance. The tilingK and tilingN parameters are determined by the compression algorithm. Currently, only the value 8 is supported.

Definition

struct LinearSparseParam{
    bool transposeA = false;
    bool transposeB = true;
    uint32_t tilingK =8;
    uint32_t tilingN = 8;
    uint8_t rsv[12] = {0};
};

Parameters

Member	Type	Default Value	Description
transposeA	bool	false	Whether to transpose matrix A. For quantization, this parameter can only be set to false for Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
transposeB	bool	true	Whether to transpose matrix B. For dequantization, this parameter can only be set to true for Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products.
tilingK	uint32_t	8	Compression parameter, which is determined by the external compression algorithm. Currently, the value can only be 8.
tilingN	uint32_t	8	Compression parameter, which is determined by the external compression algorithm. Currently, the value can only be 8.
rsv[12]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[m, k]	int8	ND	Matrix A for matrix multiplication. m must be less than or equal to 256, and k must be an integer multiple of 64 and greater than 256.
weight	[c]	int8	ND/NZ	Weight, matrix B for matrix multiplication. Weight compressed by the compression tool. The value of shape size c is greater than 0 and less than or equal to k × n.
bias	[1, n] or [n]	int32	ND	Added bias matrix n is an integer multiple of 64 and is greater than or equal to 128.
deqScale	[1, n] or [n]	uint64/int64	ND	Scale of dequantization. This parameter is input during quantization.
compressIdx	[x]	int8	ND	Compression index generated during weight compression. x is calculated as follows: x = k_index * n_index * 8 k_index = ceil(k1 / tilingK) n_index = ceil(n1 / tilingN) k1 = k / 32 n1 = n / 16 The ceil() function rounds up the final value.

Output

Parameter	Dimension	Data Type	Format	Description
output	[m, n]	float16	ND	Output tensor. The number of dimensions is the same as that of x.

Constraints

Use transposeA and transposeB to configure the original x/weight dimensions to meet the dimension relationship of matrix multiplication.
The dimension of the input weight (c)is calculated by a compression tool (such as ModelSlim), and is greater than 0 but less than k × n.
The tilingK and tilingN parameters are determined by the compression algorithm. Currently, only the value 8 is supported.

Parent topic: atb/infer_op_params.h