QuantizeAddLayerNormPass

Description

Scenario 1: Fuses the two Quantize operators or two AscendQuantV2 operators connected to the output y of AddLayerNorm into the DuaQuantizeAddLayerNorm operator.

Scenario 2: Fuses the AscendQuant or AscendQuantV2 operator connected to the output y of AddLayerNorm into the QuantizeAddLayerNorm operator.

Scenario 3: Using Atlas 350 Accelerator Card, fuses the AddLayerNorm operator and one or two Quantize or AscendQuantV2 operators connected to it into the AddLayerNormQuant operator.

Restrictions

• Scenarios involving Atlas 350 Accelerator Card
  • The quantization node must be Quantize or AscendQuantV2. Single- or dual-input quantization is supported. In dual-input quantization, the two quantization nodes must be of the same type. Only the int8 quantization type is supported.
  • The data type of the AddLayerNorm input parameter x1 can be bf16, fp16, or fp32. The data types of other input parameters must be the same as that of x1. The AddLayerNorm output does not contain mean or rstd.
  • If the quantization node is Quantize, the data type of the input parameter x must be the same as that of the input parameter x1 of AddLayerNorm. The data type of the input parameter scale must be fp32 or the same as that of the input parameter x1 of AddLayerNorm.

• Scenarios involving Atlas A2 training product/Atlas A2 inference product and Atlas A3 training product/Atlas A3 inference product
  • The number of input parameters of AddLayerNorm must be at least 5, and the parameter bias must exist.
  • In the scenario where the operator is fused into DuaQuantizeAddLayerNorm, the AddLayerNorm node must be connected to two quantization nodes. The data type of the input parameter x1 of AddLayerNorm must be bf16, and the tail axis must be 32-byte aligned.
  • In the scenario where the operator is fused into QuantizeAddLayerNorm, the AddLayerNorm node must be connected to a single quantization node. When the quantization node is AscendQuant, the data type of the input parameter x1 of AddLayerNorm cannot be fp32, and the tail axis must be 32-byte aligned.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas A3 training product/Atlas A3 inference product

Atlas 350 Accelerator Card