In this quantization mode, activations are not quantized, and only weights are quantized to 8-bit format. The per-channel quantization scheme is used.
[object Object]
Weight directory structure after quantization:
- Quantization output includes:
[object Object](weight file) and[object Object](weight description file). - The other files in the directory are required for inference, and they vary slightly by model.
The following is a partial view of [object Object] after quantization:
Quantized MatMul weights include [object Object] and [object Object] to dequantize the MatMul output.
Figure 1 Process of inference with quantized weights[object Object][object Object]
This quantization mode supports quantization of the original weights of the float16 or bfloat16 type.
Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])
Table 2 dtype and shape information after bfloat16 weight quantization (assuming that shape of the original weight is [object Object])
[object Object]