ADA Algorithm

During neural network quantization, each floating-point weight is assigned to its closest fixed-point value. However, this is not the optimal quantization policy. In this document, Adaptive Rounding (AdaRound), a better weight rounding mechanism for PTQ, is introduced. It adapts to the data and task loss and requires only a small amount of unlabeled data.

The ADA algorithm is used in the PTQ process. Since the rounding mode of the quantized weight affects model accuracy, ADA fine-tunes the rounding mode, adaptively rounding up or down each quantized data. Unlabeled data is required for training and tuning to determine different rounding modes. The following figure shows the overall process. That is, the initial weight quantization factors scale and offset are calculated using the KL divergence algorithm, and then unlabeled data is used for training to adjust the rounding mode of the weight.

The quantized weight is calculated as follows:

, where n and p are the upper and lower bounds after quantization, respectively.

The training process is to optimize the alpha parameter. The goal is to make h(alpha) equal to 0 or 1:

, where and are scaling parameters and are set to 1.1 and −0.1, respectively.

Quantize the source weight, add h(alpha) to dequantize the weight, introduce the dequantized weight to the final operator computation, and output the result with a quantization error.

The loss function is calculated as follows:

Regularization (the latter part of the loss function) is used to constrain h(alpha) to converge to 0 and 1. In the early stage of iteration, a large value of increases the impact of regularization on the overall loss, which quickly adjusts h(alpha) to the optimal solution area. In the later stage of iteration, h(alpha) is already in the optimal solution area. A small value of does not cause a sharp change in the regularization in the case of a slight parameter change. As a result, the model is more robust when parameters are fine-tuned, avoiding missing the optimal solution due to over-adjustment. This makes h(alpha) approach the optimal solution.