[object Object][object Object]

In this quantization mode, the weights and activations are quantized, and the high-precision floating point numbers are converted into 8-bit numbers to reduce the size of the model weight. Using data in int8 format for computation can reduce the computation workload of the matrix multiplication (MatMul) operator and improve the inference performance.

Weight directory structure after quantization:

• Quantized outputs: [object Object] (weight file) and [object Object] (weight description file).
• The other files in the directory are required for inference, and they vary slightly by model.

The following shows part of the content in weight description file [object Object] after quantization:

Quantized MatMul parameters include [object Object], [object Object], [object Object], and [object Object]. [object Object] and [object Object] are used to quantize the activations. MatMul uses the quantized activations and weights for computation. [object Object] and [object Object] are used to dequantize the computation result of MatMul.

Figure 1 Process of inference with quantized weights [object Object][object Object]

This quantization mode supports quantization of the original weights of the fp16 or bf16 type.

Table 1 dtype and shape information after fp16 weight quantization (assuming that the shape of the original weight is [object Object])

[object Object]undefined

Table 2 dtype and shape information after bf16 weight quantization (assuming that the shape of the original weight is [object Object])

[object Object]undefined[object Object]

You can use the tool to generate quantized weights.

The following uses Qwen2-7B as an example. After installing msModelSlim, you can run the following command to quickly generate the W8A8-quantized weights:

The preceding command is a best practice of msModelSlim. For details about more quantization parameter configurations, see the msModelSlim documentation.

[object Object]

The following uses the Qwen2-7B-W8A8-quantized weight as an example. You can run the following commands to perform a dialog test, with the inference content being "What's deep learning?" and a maximum output of 20 tokens.