IncreFlashAttentionQuantFusionPass

Description

Fuses IncreFlashAttention+AscendQuant into the IncreFlashAttention operator in the quantization scenario. The scale and offset parameters of quant are converted into the input parameters quant_scale2 and quant_offset2 of ifa.

Restrictions

• The output of IncreFlashAttention must be fp16.
• The input of AscendQuant must be fp16, and the output must be int8.
• quant_scale2 and quant_offset2 of the IncreFlashAttention operator must be empty.
• The IncreFlashAttention operator can have only one output.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card