Application Scenarios

It is usually used before the attention mechanism and between the attention and the feedforward network ffn.
Figure 1 Context of the RmsNormOperation operator
Application scenario when rstd is set to true
- Input: xDim:[8,8,8]; gammaDim:[8]; Output: rstdDim:[8, 8, 1]
  i
  
  xDimNum - gammaDimNum
  
  condition(i >= (xDimNum - gammaDimNum))
  
  rstdDimNum
  
  0
  
  2
  
  false
  
  xDim[0]
  
  1
  
  2
  
  false
  
  xDim[1]
  
  2
  
  2
  
  true
  
  1
- Input: xDim:[8,8,8]; gammaDim:[8,8]; Output: rstdDim:[8, 1, 1]
  i
  
  xDimNum - gammaDimNum
  
  condition(i >= (xDimNum - gammaDimNum))
  
  rstdDimNum
  
  0
  
  1
  
  false
  
  xDim[0]
  
  1
  
  1
  
  true
  
  1
  
  2
  
  1
  
  true
  
  1
- Input: xDim: [8, 16, 32]; gammaDim: [8, 16, 32]; Output: rstdDim: [1, 1, 1]
  i
  
  xDimNum - gammaDimNum
  
  condition(i >= (xDimNum - gammaDimNum))
  
  rstdDimNum
  
  0
  
  0
  
  true
  
  1
  
  1
  
  0
  
  true
  
  1
  
  2
  
  0
  
  true
  
  1

i	xDimNum - gammaDimNum	condition(i >= (xDimNum - gammaDimNum))	rstdDimNum
0	2	false	xDim[0]
1	2	false	xDim[1]
2	2	true	1

i	xDimNum - gammaDimNum	condition(i >= (xDimNum - gammaDimNum))	rstdDimNum
0	1	false	xDim[0]
1	1	true	1
2	1	true	1

i	condition(i >= (xDimNum - gammaDimNum))	rstdDimNum
0	true	1
1	true	1
2	true	1

Parent topic: RmsNormOperation