FusedInferAttentionScoreQuantFusionPass

Description

Fuses FusedInferAttentionScore+AscendQuant into the FusedInferAttentionScore operator in the quantization scenario. The scale and offset parameters of quant are converted into the input parameters quant_scale2 and quant_offset2 of the FIA interface.

Restrictions

• FusedInferAttentionScore supports only fp16 output.
• AscendQuant supports only fp16 input and int8 output.
• quant_scale2 and quant_offset2 of the FusedInferAttentionScore operator must be empty.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card