AddRmsNormQuantFusionPass
Description
Fuses the structure that meets the following patterns into the AddRmsNormQuant operator.
Scenario 1: Fuses the AddRmsNorm and Quantize operators that comply with the graph fusion pattern into fused operator AddRmsNormQuant. The output y of the AddRmsNorm operator is used as the first input of the Quantize operator.
Scenario 2: Fuses the AddRmsNorm, Reshape, and Quantize operators that comply with the graph fusion pattern into fused operator AddRmsNormQuant. The output y of the AddRmsNorm operator is used as the input of the Reshape operator, and the output of the Reshape operator is used as the first input of the Quantize operator.
Restrictions
- In the following products, the Quantize operator is restricted to producing int8 outputs only.
- Atlas A2 Training Series Product/Atlas 800I A2 Inference Product
- Atlas Inference Series Product
- Atlas A3 Training Series Product
- The x1 input of AddRmsNorm supports only the float16 and bfloat16 data types, and the last axis of x1's shape must be 32-byte aligned.
- The AddRmsNorm operator before fusion does not output rstd.
- The AddRmsNormQuant operator after fusion does not output y2.
- The number of elements in the inputs scales and zero_points before and after fusion must be the same as that of the input gamma. If the shape dimension of the input gamma before fusion is inconsistent with that of scales or zero_point, you are advised to use scenario 2 for fusion.
Availability
Atlas 350 Accelerator Card