AddRmsNormDynamicQuantFusionPass

Description

Fuses the AddRmsNorm and DynamicQuant operators that comply with the graph fusion pattern into fused operator AddRmsNormDynamicQuant. The output y of the AddRmsNorm operator is used as the first input of the DynamicQuant operator.

Scenario 1: single-channel mode

Scenario 2: dual-channel mode

Restrictions

• Before fusion, the input types of the AddRmsNorm and DynamicQuant operators must be the same (fp16 or bf16).
• Restrictions on the output yout of DynamicQuant:
  • Atlas A2 training product/Atlas A2 inference product: yout only supports the int8 quantization type.
  • Atlas A3 training product/Atlas A3 inference product: yout only supports the int8 quantization type.
• Restrictions on dtype:
  • If there is a smoothing coefficient, the data type of DynamicQuant's smooth_scales must be the same as that of AddRmsNorm's x1 input.
• Restrictions on shapes:
  • The shape of gamma must be 1D, and its size must match the last dimension of the inputs x1 and x2, that is, gamma.shape = [x1.shape[-1]].

• Restrictions on attr:
  • Atlas 350 Accelerator Card: dst_type can only be DT_INT8, DT_HIFLOAT8, DT_FLOAT8_E4M3FN, or DT_FLOAT8_E5M2.
  • Atlas A2 training product/Atlas A2 inference product: dst_type can only be DT_INT8.
  • Atlas A3 training product/Atlas A3 inference product: dst_type can only be DT_INT8.
  • In dual-channel mode, DynamicQuant0 must have the same dst_type as DynamicQuant1.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card