[object Object][object Object]

Flash Attention 3 (FA3) quantization is similar to attention quantization. The difference is that DeepSeek uses the MLA algorithm and the value of K RoPE changes too much, which is not suitable for quantization. Therefore, in this quantization method, the non-rope tensor of k is quantized to 8bits, and the rope tensor of k is not quantized. The currently used quantization scheme is perhead quantization. Partial quantization of k is performed to reduce the graphics memory usage of the KV cache, optimize the speed of the attention operator in the decode phase, and improve the throughput.

[object Object]

Directory structure of quantized weights after FA3 and W8A8 quantization:

• Quantization output includes: [object Object] (weight file) and [object Object] (weight description file).
• The other files in the directory are required for inference, and they vary slightly by model.

The following is a partial view of [object Object] after quantization:

Compared with the W8A8 weight quantization, description field [object Object] as well as field [object Object] and its content are added. [object Object] is used to quantize the [object Object] and [object Object] features to the INT8 type, and [object Object] is used to dequantize the [object Object] and [object Object] output to the floating-point type.

Figure 1 Inference process for FA3 weight quantization

Table 1 dtype and shape information after float16 weight quantization (assuming that shape of the original weight is [n, k])

[object Object]undefined[object Object]

1. Install .

2. Complete the required checks before running DeepSeek-V3/R1. For details, see the .

3. Go to the [object Object] directory and run the following quantization command:

   [object Object]

   The quant_model_description.json file of the FA3 quantized weights must contain the "fa_quant_type": "FAKQuant" key-value pair.

[object Object]

1. Enable the NZ format for the KV cache.

   • For pure model inference: Set [object Object] in [object Object].

   • For serving inference: Add the [object Object] field under [object Object] in [object Object] as shown below.

     [object Object]

   [object Object]

2. You can run the following commands to perform a dialog test. The inference content is "What's deep learning?".

   [object Object]