[object Object]

[object Object][object Object]undefined[object Object]

• Description: Normalizes the input data at the network layer to the [0, 1] range. The LayerNormGrad operator is a key operator used in the backpropagation phase of deep learning. It is mainly used to compute the gradient of the LayerNorm operation. The AddLayerNormGrad operator fuses the Add and LayerNormGrad operators to reduce data move-in and move-out operations.

• Formula:

  • Forward formula: (D indicates the size of the reduced axis.)

    $$x= inputx1 + inputx2$$ $$\operatorname{LayerNorm}(x)=\frac{x_i−\operatorname{E}(x)}{\sqrt{\operatorname{Var}(x)+ eps}}*gamma + beta$$ $$\operatorname{E}(x_i)=\frac{1}{D}\sum_{1}^{D}{x_i}$$ $$\operatorname{Var}(x_i)=\frac{1}{D}\sum_{1}^{D}{(x_i-\operatorname{E}(x))^2}$$

  • Backward formula:

    $$x= inputx1 + inputx2$$ $$dxOut = \sum_{j}{inputdy_i * gamma_j * \frac{{\rm d}\hat{x_j}}{{\rm d}x_i}} + dsumOptional$$ $$dgammaOut = \sum_{j}{inputdy_i * \frac{{\rm d}\hat{x_j}}{{\rm d}x_i}}$$ $$dbetaOut = \sum_{j}{inputdy_i}$$

    Where:

    • $\hat{x_j}$:

      $$\hat{x_j}=({x_i-\operatorname{E}(x)}) * {rstd}$$

    • $rstd$:

      $$rstd=\frac {1}{\sqrt{\operatorname{Var}(x)}}$$

    • $\frac{{\rm d}\hat{x_j}}{{\rm d}x_i}$:

      $$\frac{{\rm d}\hat{x_j}}{{\rm d}x_i}=(\delta_{ij} - \frac{{\rm d}\operatorname{E}(x)}{{\rm d} x_i}) * \frac{1}{\sqrt{\operatorname{Var}(x_i)}}-\frac{1}{\operatorname{Var}(x_i)} (x_j-\operatorname{E}(x))\frac{\rm d \operatorname{Var}(x_i)}{\rm dx}$$

      When i=j, $\delta_{ij}$=1; when i!=j, $\delta_{ij}$=0.

    • $\frac{{\rm d}\operatorname{E}(x)}{{\rm d}x_i}$:

      $$\frac{{\rm d}\operatorname{E}(x)}{{\rm d}x_i}=\frac{1}{D}$$

      D is the number of elements in x that participate in the mean calculation.

    • $\frac{\rm d \operatorname{Var}(x_i)}{\rm dx}$:

      $$\frac{\rm d \operatorname{Var}(x_i)}{\rm dx}=\frac{1}{D}\frac{1}{\sqrt{\operatorname{Var} (x_i)}}(x_i-\operatorname{E}(x))$$

    • Simplified $dxOut$:

      $$dxOut = rstd * ({inputdy_i * gamma_j} - \frac{1}{D} * (\sum_{j}{inputdy_i * gamma_j} + \hat {x_j} * \sum_{j}{inputdy_i * gamma_j * \hat{x_j}})) + dsumOptional$$

[object Object]

Each operator has calls. First, aclnnAddLayerNormGradGetWorkspaceSize is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, aclnnAddLayerNormGrad is called to perform computation.

[object Object]

• Parameters

  [object Object]
  • [object Object]Atlas inference products[object Object]: The data types of [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] cannot be BFLOAT16.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Function dimension

  • The supported data types are as follows:
    • [object Object]Atlas inference products[object Object]: dy, x1, x2, gamma, dsumOptional, and dxOut support FLOAT32 and FLOAT16.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: dy, x1, x2, gamma, dsumOptional, and dxOut support FLOAT32, FLOAT16, and BFLOAT16.
    • rstd, mean, dgammaOut, and dbetaOut support FLOAT32.
  • The data format can be ND.

• Description of unsupported types

  DOUBLE: The instructions do not support DOUBLE.

• Description of boundary value scenarios

  • When the input is Inf, the output is Inf.
  • When the input is NaN, the output is NaN.

• Deterministic compute:

  • aclnnAddLayerNormGrad defaults to a non-deterministic implementation. You can call aclrtCtxSetSysParamOpt to enable deterministic compute.

[object Object]

The following example is for reference only. For details, see .