文档
下载

Normalize

Applicability

Product

Supported

Atlas 350 Accelerator Card

√

Atlas A3 training product/Atlas A3 inference product

√

Atlas A2 training product/Atlas A2 inference product

√

Atlas 200I/500 A2 inference product

x

Atlas inference product AI Core

√

Atlas inference product Vector Core

x

Atlas training product

x

Function Usage

Computes the reciprocal rstd of the standard deviation of the input data with a shape of [A, R] and y based on the known mean and variance in LayerNorm. The formulas are as follows:

E and Var respectively represent the mean and variance of an input on the R axis. γ is the scaling coefficient, β is the translation coefficient, and ε is the weight coefficient for preventing division by zero.

Prototype

  • Pass the temporary space through the sharedTmpBuffer input parameter.
    1
2
    template < typename U, typename T, bool isReuseSource = false, const NormalizeConfig& config = NLCFG_NORM>
__aicore__ inline void Normalize(const LocalTensor<T>& output, const LocalTensor<float>& outputRstd, const LocalTensor<float>& inputMean, const LocalTensor<float>& inputVariance, const LocalTensor<T>& inputX, const LocalTensor<U>& gamma, const LocalTensor<U>& beta, const LocalTensor<uint8_t>& sharedTmpBuffer, const float epsilon, const NormalizePara& para)
  • Allocate the temporary space through the API framework.
    1
2
    template < typename U, typename T, bool isReuseSource = false, const NormalizeConfig& config = NLCFG_NORM>
__aicore__ inline void Normalize(const LocalTensor<T>& output, const LocalTensor<float>& outputRstd, const LocalTensor<float>& inputMean, const LocalTensor<float>& inputVariance, const LocalTensor<T>& inputX, const LocalTensor<U>& gamma, const LocalTensor<U>& beta, const float epsilon, const NormalizePara& para)

Due to the complex computation involved in the internal implementation of this API, additional temporary space is required to store intermediate variables generated during computation. The temporary space can be allocated through the API framework or passed by developers through the sharedTmpBuffer input parameter.

  • When the API framework is used for temporary space allocation, developers do not need to allocate the space, but must reserve the required size for the temporary space.
  • When the sharedTmpBuffer input parameter is used for passing the temporary space, the tensor serves as the temporary space. In this case, the API framework is not required for temporary space allocation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization.

If the API framework is used, developers must reserve the temporary space. If sharedTmpBuffer is used, developers must allocate space for the tensor. The method of obtaining the temporary space size (BufferSize) is as follows: Obtain the required maximum and minimum temporary space sizes using the GetNormalizeMaxMinTmpSize API provided in Normalize Tiling. The minimum space can ensure correct functionality, while the maximum space is used to improve performance.

Parameters

Table 1 Template parameters

Parameter

Description

U

Data type of the beta and gamma operands.

For the Atlas 350 Accelerator Card, the supported data types are half, bfloat16_t, and float.

For the Atlas A3 training product/Atlas A3 inference product, the supported data types are half and float.

For the Atlas A2 training product/Atlas A2 inference product, the supported data types are half and float.

For the Atlas inference product AI Core, the supported data types are half and float.

T

Data type of the output and inputX operands.

For the Atlas 350 Accelerator Card, the supported data types are half, bfloat16_t, and float.

For the Atlas A3 training product/Atlas A3 inference product, the supported data types are half and float.

For the Atlas A2 training product/Atlas A2 inference product, the supported data types are half and float.

For the Atlas inference product AI Core, the supported data types are half and float.

isReuseSource

This parameter is reserved. Pass the default value false.

config

A parameter used to configure the input and output information of the Normalize API. The NormalizeConfig type is defined as follows:

1
2
3
4
5
6
7
struct NormalizeConfig {
    ReducePattern reducePattern = ReducePattern::AR;
    int32_t aLength = -1;
    bool isNoBeta = false;
    bool isNoGamma = false;
    bool isOnlyOutput = false;
};
  • reducePattern: Currently, only the ReducePattern::AR mode is supported, indicating that the input inner R axis is the reduced axis.
  • aLength: Size of the input axis A. The following values are supported:
    • –1: Default value. The aLength value of API parameter para is used as the A axis size.
    • 1: Supports non-aligned transfer-out of outputRstd data and non-aligned transfer-in of inputMean and inputVariance data. If aLength is set to other values, non-aligned transfer-in or non-aligned transfer-out of the preceding inputs and outputs is not supported. The value must be the same as the aLength value of API parameter para. Note that only Atlas 350 Accelerator Card supports this value.
    • Other values: The value must be the same as the aLength value of API parameter para.
  • isNoBeta: Whether to use beta in computation.
    • false: Default value. The input beta is used in the Normalize computation.
    • true: The input beta is not used in the Normalize computation. In this case, computation related to beta in the formula is omitted.
  • isNoGamma: Whether the optional input gamma is used.
    • false: Default value. The optional input gamma is used in the Normalize computation.
    • true:The input gamma is not used in the Normalize computation. In this case, computation related to gamma in the formula is omitted.
  • isOnlyOutput: Whether to output only y but not the reciprocal rstd of the standard deviation. Currently, this parameter can only be set to false, indicating that all y and rstd results are output.
Table 2 API parameters

Parameter

Input/Output

Description

output

Output

Destination operand, with a shape of [A, R]. For details about the definition of the LocalTensor data structure, see LocalTensor.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

outputRstd

Output

Reciprocal of the standard deviation, with a shape of [A]. For details about the definition of the LocalTensor data structure, see LocalTensor.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

inputMean

Input

Mean, with a shape of [A]. For details about the definition of the LocalTensor data structure, see LocalTensor.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

inputVariance

Input

Variance, with a shape of [A]. For details about the definition of the LocalTensor data structure, see LocalTensor.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

inputX

Input

Source operand, with a shape of [A, R]. For details about the definition of the LocalTensor data structure, see LocalTensor. The data type of inputX must be the same as that of the destination operand, and the last axis length must be 32-byte aligned.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

gamma

Input

Scaling coefficient, with a shape of [R]. For details about the definition of the LocalTensor data structure, see LocalTensor. The data type precision of gamma must be greater than or equal to that of the source operand.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

beta

Input

Translation coefficient, with a shape of [R]. For details about the definition of the LocalTensor data structure, see LocalTensor. The data type precision of beta must be greater than or equal to that of the source operand.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

sharedTmpBuffer

Input

Shared buffer, which is used to store temporary data generated during internal API computation. This enables developers to manage the sharedTmpBuffer space and reuse the buffer after calling the API, so that the buffer is not repeatedly allocated or deallocated, improving the flexibility and buffer utilization. For details about how to obtain the size of the shared buffer, see Normalize Tiling.

The type is LocalTensor, and TPosition can be VECIN, VECCALC, or VECOUT.

epsilon

Input

Weight coefficient for preventing division by zero.

para

Input

Parameter information required for the Normalize computation. The NormalizePara type is defined as follows:

1
2
3
4
5
struct NormalizePara {
    uint32_t aLength;
    uint32_t rLength;
    uint32_t rLengthWithPadding;
};
  • aLength: Specifies the length of inputX along the A axis.
  • rLength: Specifies the length of inputX along the R axis.
  • rLengthWithPadding: Specifies the 32-byte aligned length of inputX along the R axis.

Returns

None

Constraints

  • For details about the operand address alignment requirements, see General Address Alignment Restrictions.
  • The data type precision of the scale coefficient gamma and translation coefficient beta must be greater than or equal to that of the source operand inputX. For example, if the data type of inputX is half, the data types of gamma and beta can be half or float, and their data type precision is not lower than that of inputX.For example, for the Atlas 350 Accelerator Card, if the data type of inputX is bfloat16_t, the data types of gamma and beta can be bfloat16_t or float, and their data type precision is not lower than that of inputX.
  • The tensor space of src and dst cannot be reused.
  • The input must be in ND format.
  • The R axis cannot be split.

Examples

For a complete call example, see Normalize operator sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// yLocal: output normalized result y, with shape [A, R]
// rstdLocal: output reciprocal of the standard deviation (1 / sqrt(variance + epsilon)), with shape [A]
// meanLocal: input mean value, with shape [A]
// varianceLocal: input variance, with shape [A]
// xLocal: input data X, with shape [A, R], with the same data type as output
// gammaLocal: scaling parameter gamma, with shape [R]
// betaLocal: translation parameter beta, with shape [R]
// epsilon: coefficient for preventing division by zero
// para: NormalizePara structure that contains the A and R dimension information
// config: Normalize configuration parameters, specifying whether to skip gamma/beta and reduce mode

constexpr AscendC::NormalizeConfig CONFIG {
    .reducePattern = AscendC::ReducePattern::AR,
    .aLength = -1,
    .isNoBeta = isNoBeta,
    .isNoGamma = isNoGamma,
    .isOnlyOutput = false
};

// Use the Normalize API to perform layer normalization computation.
AscendC::Normalize<DTYPE_Y, DTYPE_X, false, CONFIG>(
    yLocal,          // Output: normalized result y, with shape [A, R]
    rstdLocal,       // Output: reciprocal of the standard deviation (rstd), with shape [A]
    meanLocal,       // Input: mean, with shape [A]
    varianceLocal,   // Input: variance, with shape [A]
    xLocal,          // Input: original data X, with shape [A, R]
    gammaLocal,      // Input: scaling coefficient γ, with shape [R]
    betaLocal,       // Input: translation coefficient β, with shape [R]
    epsilon,         // Input: coefficient ε for preventing division by zero
    para             // Input: tiling parameters, including aLength, rLength, and rLengthWithPadding
);
The following is an example:
Input (srcLocal, shape: [8, 8]):
[  0.  1.  2.  3.  4.  5.  6.  7.
   8.  9. 10. 11. 12. 13. 14. 15.
  16. 17. 18. 19. 20. 21. 22. 23.
  24. 25. 26. 27. 28. 29. 30. 31.
  32. 33. 34. 35. 36. 37. 38. 39.
  40. 41. 42. 43. 44. 45. 46. 47.
  48. 49. 50. 51. 52. 53. 54. 55.
  56. 57. 58. 59. 60. 61. 62. 63. ]
Input (meanLocal, shape: [8]):
[ 0. 1. 2. 3. 4. 5. 6. 7. ]
Input (varianceLocal, shape: [8]):
[ 0. 1. 2. 3. 4. 5. 6. 7. ]
Input (gammaLocal, shape: [8]):
[ 1. 1. 1. 1. 1. 1. 1. 1. ]
Input (betaLocal, shape: [8]):
[ 1. 1. 1. 1. 1. 1. 1. 1. ]
Output (yLocal):
[ 1.0 32.622772 64.245544 95.868324 127.4911 159.11388 190.73665 222.35942 
  7.996503 8.996003 9.995503 10.995004 11.994504 12.994005 13.9935055 14.993006 
  10.897021 11.603951 12.310882 13.017812 13.724742 14.431672 15.138602 15.845532 
  13.122336 13.699591 14.276845 14.854099 15.431353 16.008606 16.585861 17.163115 
  14.998251 15.498188 15.998126 16.498064 16.998001 17.497938 17.997875 18.497814 
  16.65091 17.09808 17.545248 17.992416 18.439585 18.886755 19.333923 19.781092 
  18.144999 18.553213 18.961428 19.369642 19.777857 20.186071 20.594284 21.002499 
  19.518936 19.896873 20.27481 20.652748 21.030685 21.408623 21.78656 22.164497 ]
Output (rstdLocal):
[ 31.622774    0.9995004   0.7069301   0.5772541   0.49993753  0.44716886  0.40821427  0.37793747 ]
Parent topic: Normalization
我们使用cookie来确保您的高速浏览体验。继续浏览本站，即表示您同意我们使用cookie。查看详情