FusedInferAttentionScoreQuantFusionPass

Description

Fuses FusedInferAttentionScore+AscendQuant into the FusedInferAttentionScore operator in quantization use cases. The scale and offset parameters of quant are converted into the input parameters quant_scale2 and quant_offset2 of FIA.

Constraints

• FusedInferAttentionScore supports only fp16 output.
• AscendQuant supports only fp16 input and int8 output.
• quant_scale2 and quant_offset2 of the FusedInferAttentionScore operator must be empty.

Applicable Products

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Ascend 950PR/Ascend 950DT