ScopeLayerNormPass
Description
Fuses the layernorm/batchnorm and layernorm/moments scopes generated by tf.layernorm into a LayerNorm operator.
Scope Details
batchnorm unfolded:
moments unfolded:
Fusion Mapping
When there are Cast nodes, the input of the first Cast node is used as the first input x of the fused operator.
The input gamma of the Mul node is used as the second input gamma of the fused operator.
The input beta of the last Add node is used as the third input beta of the fused operator.
The fourth begin_norm_axis uses the default value 1.
The fifth begin_param_axis uses the default value –1.
Applicable Network
BERT
Fusion Pattern Type
General fusion pattern
Parent topic: Fusion Patterns