[object Object]

[object Object]

[object Object]

Anti-Outlier is used to solve the problem of accuracy drop caused by abnormal activation value distribution (outlier) in model quantization. During LLM quantization, if there are outliers with extremely large values in the activation values, the quantization range (scale) is enlarged. As a result, the quantization resolution of most normal values is reduced, which severely affects the model accuracy. This technology smooths or suppresses outliers to effectively improve data distribution and ensure that the quantized model can still maintain high inference accuracy.

[object Object]

The following shows part of the content in weight description file [object Object] after the W8A8 + Anti-Outlier + PDMIX quantization:

Currently, mainstream open-source LLMs (such as Llama and Qwen) use RmsNorm as [object Object] and [object Object]. When the asymmetric anti-outlier algorithm is enabled, an extra bias item (referred to as [object Object]) is introduced.

To ensure calculation equivalence, the execution logic of the algorithm is as follows:

1. Norm layer introduction: When the Norm operation is performed, [object Object] is added to the weights.
2. Linear layer offset: Before the subsequent linear layer calculation, the corresponding [object Object] is subtracted.

In the actual model weight, the offset tensor of [object Object] at the linear layer is fused in different ways based on the quantization scenario.

• Per-tensor scenario: Directly fused into the quantization bias [object Object] at the linear layer.
  • The preceding [object Object] is an example.
• Per-token scenario: Represented by the common bias [object Object] at the linear layer.
  • The preceding [object Object] is an example.

[object Object]

Performance Optimization Suggestions In the PDMIX scenario (per-token quantization is used in the P phase), the anti-outlier bias has little impact on the quantization accuracy. To improve performance, you can remove the bias from both the norm layer and the linear layer while ensuring equivalence, thereby reducing the overhead of one Add operation.

Figure 1 Inference process for quantizing weights

Table 1 dtype and shape information of some layers after quantization (assuming original weight shape is [object Object])

[object Object]undefined[object Object]

You can use the tool to generate quantized weights.

The following uses Qwen3-14B as an example. After installing msModelSlim, you can run the following command to quickly generate the W8A8PDMIX quantization weights with outlier suppression:

When the preceding command is executed, the best practice of msModelSlim is used for quantization by default. For details about more quantization parameter configurations, see the msModelSlim documentation.

[object Object]

Using the Qwen3-14B-W8A8PDMIX weights as an example, you can run the following command to perform a dialog test. The inference content is "What's deep learning?" and the maximum number of output tokens is 20.