ScopeLayerNormGradPass
Description
Fuses the backward scope of tf.layernorm into a LayerNormGrad operator.
Scope Details
mul_grad unfolded:
sub_grad unfolded:
Result Operator Prototype
LayerNormGrad. For details, see Operator Acceleration Library API Reference.
Fusion Mapping
The backward input of LayerNorm is used as the first input dy of the fused operator.
The forward input of LayerNorm is used as the second input x of the fused operator.
The third forward output variance is used as the third backward input variance.
The second forward output mean is used as the third backward input mean.
The second forward input gamma is used as the fourth backward input gamma.
The first backward output connects to the output of the last addN node in the backward graph.
The second backward output gamma_backprop connects to the output of the Mul node in mul_grad that will connect to a cast node.
The third backward output beta_backprop connects to the output of the Sum node in sub_grad that will connect to a cast node.
Applicable Network
BERT
Fusion Pattern Type
General fusion pattern