Dequantization Fusion

Description

Quantized k and v can be passed to reduce the graphics memory usage. Because the input q is not quantized, the process is also called fake quantization.

How to Enable

Set quantType to TYPE_DEQUANT_FUSION.

When the preceding parameter is enabled, the hasQuantOffset parameter can be used to specify whether the input quantized KV requires the corresponding offset.

Constraints

Not supported by Atlas inference products.