LayerNormFusionPass

Description

Identifies and fuses small operators such as ReduceMean and SquaredDifference into the LayerNorm operator.

• Scenario 1

  Before: After:

• Scenario 2

Before: After:

Restrictions

• The dynamic shape scenario is not supported.
• Restrictions on four input parameters:
  • InputTensor: The data format is ND, NCHW, or NHWC. The shape dimension must be greater than or equal to 1, and the value of the last dimension of the shape cannot be 1.
  • Const_0: 1D. The length of the shape must be 1.
  • Const_1: 1D. The length of the shape is equal to the length of the last dimension of the InputTensor shape.
  • Const_2: 1D. The length of the shape is equal to the length of the last dimension of the InputTensor shape.
• Input ReduceMean restrictions:
  • The axes parameters of the two ReduceMean operators must be the same and must be the same as the last dimension of InputTensor.
  • The keep_dims parameters of the two ReduceMean operators must be true.
• Input Sub restrictions:
  • The first parameter must be InputTensor.
  • The second parameter is the result of ReduceMean_0 and cannot be swapped.
• Data type restrictions:

  Only FLOAT32 and FLOAT16 are supported.

Availability

Atlas A2 training product/Atlas A2 inference product