This quantization method quantizes q, k, and v to 8-bit values, reducing the graphics memory usage of the KV cache, optimizing the speed of the attention operator in the decode phase, and improving throughput.
[object Object]
Directory structure of quantized weights after attention and W8A8 quantization:
- The quantized output contains the weight file
[object Object]and the weight description file[object Object]. - The other files in the directory are required for inference, and they vary slightly by model.
The following is a partial view of [object Object] after quantization:
Compared with the W8A8 weight quantization, description field [object Object] as well as field [object Object] and its content are added. [object Object] is used to quantize the [object Object], [object Object], and [object Object] features to the INT8 type, and [object Object] is used to dequantize the [object Object], [object Object], and [object Object] output to the floating-point type.
Figure 1 Inference process for quantizing weights
Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])
Table 2 dtype and shape information after bfloat16 weight quantization (assuming that shape of the original weight is [object Object])