Dequantization Fusion

Description

Quantized k and v can be passed to reduce the graphics memory usage. Because the input q is not quantized, the process is also called fake-quantization.

How to Enable

Set quantType to TYPE_DEQUANT_FUSION.

When the preceding parameter is enabled, the hasQuantOffset parameter can be used to specify whether the input quantized KV requires the corresponding offset.

Constraints

Not supported by the Atlas inference products.