Hybrid quantization is to use different quantization modes for different levels of a model. W4A8 hybrid quantization of DeepSeek R1/V3: The first three layers of MLP use W8A8 dynamic quantization, the MLA and shared expert layers use W8A8 quantization, and the routing expert layer uses W4A8 dynamic quantization. W4A8 dynamic quantization uses per-channel and per-group to perform 4-bit quantization on weights and 8-bit quantization on activations.
[object Object]
Weight directory structure after quantization:
- Quantization output includes:
[object Object](weight file) and[object Object](weight description file). - The other files in the directory are required for inference, and they vary slightly by model.
The following is a partial view of [object Object] after quantization:
Quantized MatMul weights now include [object Object], [object Object], and [object Object] to dequantize the MatMul computation results.
Figure 1 Process of inference with quantized weights [object Object][object Object]
This quantization mode supports quantization of the original weights of the float16 or bfloat16 type.
Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])
Table 2 dtype and shape information after bfloat16 weight quantization (assuming that shape of the original weight is [object Object])
[object Object]