In this quantization method, the k cache and v cache are quantized to 8-bit values, reducing the graphics memory usage of the KV cache. In scenarios where the graphics memory is limited (for example, in long-sequence scenarios), the number of recomputations can be reduced to improve the throughput.
[object Object]
Directory structure of quantized weights after KV Cache INT8 and W8A8 quantization:
- Quantized outputs include:
[object Object](weight file) and[object Object](weight description file). - The other files in the directory are required for inference, and they vary slightly by model.
The following shows part of the quantized weight description file [object Object]:
Compared with W8A8-quantized weights, the following are added: [object Object] description field, the [object Object] quantization scaling factor weight file for KV linear activations, and the [object Object] quantization offset weight file for KV linear activations. During inference, these two weights are used to derive: [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object]. [object Object] and [object Object] quantize the [object Object] and [object Object] features to [object Object]. [object Object] and [object Object] dequantize the outputs of paged attention to the floating-point format.
Figure 1 Process of inference with quantized weights[object Object][object Object]
Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [object Object])
Table 2 dtype and shape information after bfloat16 weight quantization (assuming that shape of the original weight is [object Object])