Compression Concepts

Indicates the data type after quantization.

Atlas 350 Accelerator Card: The supported quantized data types are HIFloat8 (HiF8), FLOAT8 (float8), MXFP8, MXFP4, FP4_E2M1, FP4_E1M2, INT8, and INT4.

PTQ refers to quantization executed after the model training is complete. Weights of the trained model are quantized from floating-point values to low-bit integers, and activations are calibrated and quantized during inference with a small amount of calibration data to minimize precision loss introduced by quantization. PTQ is easy to use and requires only a small calibration dataset, which is applicable to scenarios where ease of use and resource saving take priority.

In general, after model training, the weights are determined. Therefore, the quantization parameters of a weight can be calculated offline based on the determinate weight value. In contrast, activations are input online. As such, the accurate value ranges are hard to be obtained and a small representative dataset is required to simulate the distributions of online activations. To obtain the quantization parameters of an activation, use the said dataset to run forward passes for the intermediate floating-point results, and calculate the quantization parameters of the activation offline based on the results. The following figure shows the working principles.

Figure 2 PTQ principles

QAT acts by including quantization in the retraining process to address the effects of quantization on the model accuracy. QAT, which generally requires a complete training dataset, emulates the errors incurred by quantization in forward passes by introducing fake-quantization (quantizing floating-point numbers to fixed-point ones, and then dequantizing them to floating-point ones) in the training process, and updates weights in training on top of the said errors, ensuring less accuracy drop in quantization.

QAT is often better than PTQ for model accuracy, but it is more time-consuming and requires a complete training dataset.

The following figure shows the working principles.

Figure 3 QAT principles

Currently, the AI processor supports symmetric and asymmetric quantization for activations, and only symmetric quantization for weights. (Quantization can be classified into symmetric quantization and asymmetric quantization based on whether the data center point is 0 after quantization. For details about the quantization algorithm principles, see Quantization Algorithm Principles.)

• Weight quantization refers to quantizing weights into lower-bit formats according to the distribution of weight values.
  • In PTQ, the weight quantization is conducted offline. Weights are read directly from your inference model, and are quantized using the quantization algorithm. After that, the quantized weights are written back to your model for activation quantization.
  • In QAT, the original floating-point weights are quantized during the forward pass, and the resulting quantized weights are used for forward computation. In the backward pass, the received gradients are applied to update the original floating-point weights.
• Activation quantization, also referred to as data quantization, means quantizing input activations into low-bit formats based on their value distribution. As the value distribution of each layer is unknown and wide-ranging until the forward pass (inference or training), activation quantization must be performed during inference or training.
  • In PTQ, online quantization is adopted. The inference model is modified by inserting bypass quantization nodes at target layers to collect input activations, and the quantization factors scale and offset are obtained via calibration. A small dataset is typically used during inference to approximate the value distribution of the full dataset, featuring simple operation and fast execution.
  • In QAT, the quantization factors scale and offset for activations are trainable parameters. During the forward pass, activations are quantized using the corresponding scale and offset. In the backward pass, backward gradients are utilized to calculate and update these scale and offset values.

The test dataset is a dataset subset for the final test of model accuracy.

Calibration refers to the process of performing forward inference to obtain data quantization factors in PTQ.

The calibration dataset is used for forward inference in PTQ. Its distribution should represent that of the full dataset. When selecting the calibration dataset, ensure it is representative; a subset of the test dataset is recommended. If the calibration dataset does not match the model or lacks representativeness, the quantization factors calculated from it will perform poorly on the full dataset, resulting in large quantization loss and low post-quantization accuracy.

The training dataset is a subset of the user's training network dataset, used for model training.

Quantization factors are the parameters used to quantize floating-point values to integers. The factors include scale and offset. The formula for quantizing floating-point values to integers (using INT8 as an example) is as follows:

• scale: scaling factor for floating-point values. This parameter is further divided into:
  • scale_d: scale factor for activation quantization. Only uniform activation quantization is supported.
  • scale_w: scale factor for weight quantization. Scalar (uniform quantization for weights of the current layer) and vector (channel-wise quantization for weights of the current layer) modes are supported. For more details about the parameters, see Record Files (PyTorch is used as an example. For details, see the record files of specific frameworks.)
• offset: quantization factor, indicating the offset. This parameter is further divided into:
  • offset_d: offset factor for activation quantization. Only uniform activation quantization is supported.
  • offset_w: offset factor for weight quantization. Similar to scale_w, this parameter supports both scalar and vector modes, and its dimension must be the same as that of scale_w. For more details about the parameters, see Record Files (PyTorch is used as an example. For details, see the record files of specific frameworks.)

Model outputs vary at different precision levels. A model generally delivers better inference accuracy in high precision. Quantization, which reduces the precision of a model or certain layers, may hurt the model's inference accuracy. To quantify the impact, the concept of quantization sensitivity is introduced.

Quantization sensitivity is used to measure the degree to which network models or quantizable layers are affected by quantization. Quantization sensitivity is calculated by comparing the output discrepancies of the entire network or individual layers before and after quantization. Common metrics include Mean Square Error (MSE) and cosine similarity.

The floating-point computation amount of a model layer is denoted as Flops. Bit complexity (bitops) combines Flops with data precision, describing the computational resource differences across different precision levels (such as float32, float16, INT8, and INT4).

The calculation formula is as follows, where Flops indicates the floating-point calculation amount, act_bit indicates the activation precision, and wts_bit indicates the weight precision.

Quantization granularity refers to the quantization computation levels applied to different input tensors of matrix operators such as MatMul in a neural network. Common quantization granularities are as follows.

• The variables m, n, and k indicate the sizes of different axes of a tensor.
• The left matrix and right matrix refer to the two input tensors used for matrix multiplication computation in the cube operator. Generally, the left matrix represents activation, and the right matrix represents weight. Verify against your actual implementation.

• Per-channel quantization

  Shortened as C quantization. This scheme quantizes the right matrix, with independent quantization parameters assigned to each channel. Assume that the shape of the right matrix is (k, n), where k is the reduce axis. The shape of the generated quantization parameter is (n, ).

  Figure 4 Per-channel quantization
  
• Per-group quantization

  Shortened as G quantization. This scheme supports quantization for either the left or right matrix (only the right matrix is supported in AMCT). Data is grouped along the reduce axis, with each group using independent quantization parameters.

  Assume that the shape of the right matrix is (k, n), where k is the reduce axis. If data is grouped along the k axis with a group size of gs, the shape of the generated quantization parameters is (k/gs, n).

  Figure 5 Per-group quantization
  
• Per-tensor quantization

  Shortened as T quantization. This scheme can quantize either the left or right matrix, where each tensor uses a shared quantization parameter.

  Assume that the shape of the left matrix is (m, k) and the shape of the right matrix is (k, n), where k is the reduce axis. The shape of the generated quantization parameter is (1, ).

  Figure 6 Per-tensor quantization
  
• Per-token quantization

  Shortened as K quantization. This scheme quantizes the left matrix, with independent quantization parameters assigned to each token.

  Assume that the shape of the left matrix is (m, k), where k is the reduce axis. The shape of the generated quantization parameters is (m, ).

  Figure 7 Per-token quantization