AddRmsNormDynamicQuantFusionPass

Description

Fuses the AddRmsNorm and DynamicQuant operators that comply with the graph fusion pattern into fused operator AddRmsNormDynamicQuant. The output y of the AddRmsNorm operator is used as the first input of the DynamicQuant operator.

Pattern 1: single-channel

Pattern 2: dual-channel

Constraints

• Before fusion, the input types of the AddRmsNorm and DynamicQuant operators must be the same (fp16 or bf16).
• For the output yout of DynamicQuant:
  • Atlas A2 training products/Atlas A2 inference products: yout supports only the int8 quantization type.
  • Atlas A3 training products/Atlas A3 inference products: yout supports only the int8 quantization type.
• For dtype:
  • If there is a smoothing coefficient, the data type of DynamicQuant's smooth_scales must be the same as that of AddRmsNorm's x1 input.
• For shapes:
  • The shape of gamma must be 1D, and its size must match the last dimension of the inputs x1 and x2, that is, gamma.shape = [x1.shape[-1]].

• For attr:
  • Ascend 950PR/Ascend 950DT: dst_type supports only DT_INT8, DT_HIFLOAT8, DT_FLOAT8_E4M3FN and DT_FLOAT8_E5M2.
  • Atlas A2 training products/Atlas A2 inference products: dst_type supports only DT_INT8.
  • Atlas A3 training products/Atlas A3 inference products: dst_type supports only DT_INT8.
  • In dual-channel pattern, DynamicQuant0 must have the same dst_type as DynamicQuant1.

Applicable Products

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Ascend 950PR/Ascend 950DT