Functions
RmsNorm
- Non-quantization
- Set rstd to true to train the rmsnormforward operator.
- The default value of precisionMode is HIGH_PRECISION_MODE. The float type is used for intermediate calculation. The value HIGH_PERFORMANCE_MODE means that the float16 type is used for the gamma (weight) multiplication.
- The default value of modelType is LLAMA_MODEL, indicating that the rmsnorm formula of the LLMA model is used. The value GEMMA_MODEL means that the rmsnorm formula of the gemma model is used. The difference is that before multiplication by gamma (weight), multiplication is performed on gamma + 1.
- Quantization
- rmsnorm quantization.
- The default value of dynamicQuantType is DYNAMIC_QUANT_UNDEFINED (normal quantization). DYNAMIC_QUANT_SYMMETRIC is also supported.
Fused Add-RMSNorm (PreNorm/PostNorm): Addition of Input x and Residual Prior to RMSNorm
- Non-quantization
- PreNorm returns the result of add+residual, and PostNorm does not return the result.
- hasBias: If the value is true, the bias tensor can be input. If the value is false, the bias tensor is not input.
- Quantization
Only Prenorm supports this function.
Parent topic: RmsNormOperation