DeepNorm

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	x
Atlas inference product AI Core	√
Atlas inference product Vector Core	x
Atlas training product	x

Function Usage

During the training process of a deep neural network, DeepNorm can be used as a replacement for LayerNorm normalization in order to improve the stability of Transformers by expanding residual connections.

This API applies DeepNorm normalization to input data with a shape size of [B, S, H]. The calculation formula is as follows:

DeepNorm(x) = LayerNorm(α x X + SubLayer(X))

SubLayer(X) typically refers to a sub-layer in the DeepNorm model, and is used to implement the self-attention mechanism. This API will be passed as an input tensor.

For details about the formula of LayerNorm, see LayerNorm.

Prototype

Pass the temporary space through the sharedTmpBuffer input parameter.

template <typename T, bool isReuseSrc = false, bool isBasicBlock = false>
__aicore__ inline void DeepNorm(const LocalTensor<T>& dstLocal, const LocalTensor<T>& meanLocal, const LocalTensor<T>& rstdLocal, const LocalTensor<T>& srcLocal, const LocalTensor<T>& gxLocal, const LocalTensor<T>& betaLocal, const LocalTensor<T>& gammaLocal, const LocalTensor<uint8_t>& sharedTmpBuffer, const T alpha, const T epsilon, DeepNormTiling& tiling)

Allocate the temporary space through the API framework.

template <typename T, bool isReuseSrc = false, bool isBasicBlock = false>
__aicore__ inline void DeepNorm(const LocalTensor<T>& dstLocal, const LocalTensor<T>& meanLocal, const LocalTensor<T>& rstdLocal, const LocalTensor<T>& srcLocal, const LocalTensor<T>& gxLocal, const LocalTensor<T>& betaLocal, const LocalTensor<T>& gammaLocal, const T alpha, const T epsilon, DeepNormTiling& tiling)

Parameters

**Table 1** Template parameters
Parameter	Description
T	Data type of the operand. For the Atlas 350 Accelerator Card, the supported data types are half and float. For the Atlas A3 training product/Atlas A3 inference product, the supported data types are half and float. For the Atlas A2 training product/Atlas A2 inference product, the supported data types are half and float. For the Atlas inference product AI Core, the supported data types are half and float.
isReuseSrc	Whether the source operand can be modified. The default value is false. If developers allow the source operand to be modified, enable this parameter, to reduce memory space usage. If this parameter is set to true, the buffer space of srcLocal is reused during internal computation of this API to save the buffer space. If this parameter is set to false, the buffer space of srcLocal is not reused during internal computation of this API. This parameter can be enabled for float data inputs but cannot be enabled for half data inputs. For details about how to use isReuseSrc, see More Examples.
isBasicBlock	If the shape information of srcTensor meets the basic block requirements, this parameter can be enabled to improve performance. By default, this parameter is disabled. For basic blocks, the shape of srcTensor must meet the following requirements: The length of the last axis (H axis) is a multiple of 64 but less than 2040. The length (*BS**) of a non-last axis is a multiple of 8.

**Table 2** API parameters
Parameter	Input/Output	Description
dstLocal	Output	Destination operand, with a shape of [B, S, H]. The length of H cannot exceed 2040. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
meanLocal	Output	Mean, destination operand, with a shape of [B, S]. The data type of meanLocal must be the same as that of dstLocal. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
rstdLocal	Output	Variance, destination operand, with a shape of [B, S]. The data type of rstdLocal must be the same as that of dstLocal. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
srcLocal	Input	Source operand, with a shape of [B, S, H]. The data type of srcLocal must be the same as that of the destination operand, and the last axis length must be 32-byte aligned. The length of H cannot exceed 2040. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
gxLocal	Input	Source operand, with a shape of [B, S, H]. The data type of gxLocal must be the same as that of the destination operand, and the last axis length must be 32-byte aligned. The length of H cannot exceed 2040. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. This parameter corresponds to the calculation result of SubLayer(X) in the formula.
betaLocal	Input	Source operand, with a shape of [H]. The data type of betaLocal must be the same as that of the destination operand, and the length must be 32-byte aligned. The length of H cannot exceed 2040. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
gammaLocal	Input	Source operand, with a shape of [H]. The data type of gammaLocal must be the same as that of the destination operand, and the length must be 32-byte aligned. The length of H cannot exceed 2040. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.
sharedTmpBuffer	Input	This parameter is used to store intermediate variables during complex internal API computation and is provided by developers. The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT. For details about how to obtain the temporary space size (BufferSize), see DeepNorm Tiling.
alpha	Input	Weight coefficient. The data type must be the same as that of the destination operand.
epsilon	Input	Weight coefficient, which is used to prevent division by zero errors. The data type must be the same as that of the destination operand.
tiling	Input	Tiling information required for DeepNorm computation. For details about how to obtain the tiling information, see DeepNorm Tiling.

Returns

None

Constraints

For details about the operand address alignment requirements, see General Address Alignment Restrictions.

When the isReuseSrc template parameter is set to false, the tensor space of srcLocal and dstLocal cannot be reused.
The input shape must be in ND format.
If the input data does not meet the alignment requirements, you need to pad the data. The padded data should be set to 0 to prevent abnormal values from affecting network computation.

Examples

For a complete call example, see DeepNorm operator sample.

// dstLocal: tensor for storing the DeepNorm computation result
// meanLocal: output mean tensor
// rstdLocal: output variance tensor
// srcLocal: input main data X, with shape [B, S, H]
// gxLocal: output of SubLayer (X)
// betaLocal: bias coefficient β of LayerNorm, with shape [H]
// gammaLocal: scaling coefficient γ of LayerNorm, with shape [H]
// alpha: scaling coefficient α of the residual connection
// epsilon: coefficient ε for preventing division by zero
// tiling: tiling information, including parameters such as dimensions and blocks

// Use the DeepNorm API to implement DeepNorm(x) = LayerNorm(α * X + SubLayer(X)).
// If the length of the last axis (H) does not exceed 2040 and is a multiple of 64, and the length of the non-last axes (B*S) is a multiple of 8, you can set isBasicBlock to true to improve performance.
// If the data type T is float and srcLocal can be modified, you can set isReuseSrc to true to reuse the srcLocal memory space to save memory.
AscendC::DeepNorm<T, isReuseSrc, isBasicBlock>(
    dstLocal,     // Output: normalized result
    meanLocal,    // Output: mean value
    rstdLocal,    // Output: reciprocal of the standard deviation (rstd)
    srcLocal,     // Input: original input X
    gxLocal,      // Input: sublayer output SubLayer(X)
    betaLocal,    // Input: bias coefficient β of LayerNorm
    gammaLocal,   // Input: scaling coefficient γ of LayerNorm
    alpha,        // Input: residual path scaling coefficient α
    epsilon,      // Input: coefficient ε for preventing division by zero
    tiling        // Input: tiling information
);

The following is an example:

Input (srcLocal, shape: [4, 2, 8]):
[  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
  16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
  32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
  48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 ]
Input (gxLocal, shape: [4, 2, 8]):
[  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
  16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
  32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
  48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 ]
Input (gammaLocal, shape: [4]):
[ 0 1 2 3 4 5 6 7 ]
Input (betaLocal, shape: [4]):
[ 0 1 2 3 4 5 6 7 ]
Output (dstLocal):
[ 0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958
  0. 1.0439204 2.0527046 3.0263522 3.9648638 4.868239 5.7364774 6.56958 ]
Output (meanLocal):
[ -15.75 -51.75 -87.75 -123.75 -159.75 -195.75 -231.75 -267.75 ]
Output (rstdLocal):
[ 106.3125 106.3125 106.3125 106.3125 106.3125 106.3125 106.3125 106.3125 ]

Parent topic: Normalization