AWQ

Activation-aware weight quantization (AWQ) is a PTQ method for foundation models. Using this method can significantly reduce memory usage and improve inference speed while maintaining model accuracy. AWQ finds that not all weights are equally sensitive to quantization errors. Protecting only 1% salient weights can greatly reduce the quantization error. With this algorithm, a small amount of calibration data is used to search for the optimal scale factor within the preset range using grid search. The significant weights are enlarged and then quantized to reduce errors by narrowing the quantization interval.

For details about the algorithm, click here.