Description

Normalizes the root mean square (RMS) of data to avoid the use of mean values. RMSNorm is a normalization method. RMSNorm adjusts the mean and variance of data to optimize the performance of deep learning models. RMSNorm not only simplifies calculation, but also improves the convergence speed and stability of the model.

Currently, three normalization operations are supported: Norm, PreNorm, and PostNorm.

Figure 1 NORM