AddLayerNormDynamicQuantFusionPass

Description

Fuses AddLayerNorm and one or two downstream DynamicQuant operators into the AddLayerNormQuant operator.

There may be one or two structures in the dashed-line box.

When the optional smooth input of DynamicQuant exists, it will be placed in the position corresponding to the scale input of AddLayerNormQuant.

The smooth operation of DynamicQuant can also be expressed by Mul. In this scenario, the optional smooth input of DynamicQuant must be empty.

Restrictions

• Restrictions on dtype:
  • For Atlas 350 Accelerator Card, AddLayerNorm supports the x1 input in three data types: FP16, BF16, and FP32.
  • For Atlas A2 training product/Atlas A2 inference product, AddLayerNorm supports the x1 input in two data types: FP16 and BF16.
  • The data types of inputs x2, gamma, beta, and bias of AddLayerNorm must be the same as that of x1.
  • The input x of DynamicQuant must be of the same data type as x1 of AddLayerNorm.
  • For Atlas 350 Accelerator Card, if DynamicQuant has the smooth parameter, the data type of smooth must be the same as that of x1 of AddLayerNorm.

• Restrictions on the graph structure:
  • AddLayerNorm can have only one or two fan-outs for its output 0, connecting to a Mul node or a DynamicQuant node.
  • When AddLayerNorm has two fan-outs for its output 0, the two downstream nodes must be both Mul nodes or both DynamicQuant nodes.
  • When a downstream node of output 0 is Mul, it must have exactly one downstream DynamicQuant node, which must not contain the optional smooth input.

Availability

Atlas A2 training product/Atlas A2 inference product

Atlas 350 Accelerator Card