PDMIX quantization uses different quantization modes in the prefill and decode phases of model inference.
Table 1 PDMIX quantization
Weight directory structure after quantization:
- Quantized outputs:
[object Object](weight file) and[object Object](weight description file). - The other files in the directory are required for inference, and they vary slightly by model.
The following shows part of the content in weight description file [object Object] after quantization:
Compared with the W8A8 weight quantization, [object Object] and [object Object] are added to dequantize the Matmul computation result.
The inference process of weight quantization is the same as that of W8A8 quantization.
This quantization mode supports quantization of the original weights of the bfloat16 type.
Table 2 dtype and shape information after bfloat16 weight quantization (assuming that shape of the original weight is [object Object])
[object Object]
You can use the tool to generate quantized weights.
The following uses Qwen3-14B as an example. After installing msModelSlim, you can run the following command to quickly generate the W8A8PDMIX quantization weights:
The preceding command is a best practice of msModelSlim. For details about more quantization parameter configurations, see the msModelSlim documentation.
The following uses Qwen3-14B-W8A8PDMIX weights as an example. You can run the following commands to perform a dialog test, with the inference content being "What's deep learning?" and a maximum output of 20 tokens.