Functions
RmsNorm
- Non-quantization
- Set rstd to true to train the rmsnormforward operator.
- precisionMode defaults to HIGH_PRECISION_MODE, and the intermediate computation uses the float type. The value HIGH_PERFORMANCE_MODE means that the float16 type is used for the gamma (weight) multiplication.
- modelType defaults to LLAMA_MODEL, indicating that the rmsnorm formula of the LLaMA model is used. The value GEMMA_MODEL means that the rmsnorm formula of the gemma model is used. The difference is that before multiplication by gamma (weight), multiplication is performed on gamma + 1.
- Quantization
- RmsNorm quantization.
- dynamicQuantType defaults to DYNAMIC_QUANT_UNDEFINED (normal quantization). DYNAMIC_QUANT_SYMMETRIC is also supported.
Fused Add and RmsNorm (PreNorm/PostNorm): Addition of Input x and Residual Prior to RmsNorm
- Non-quantization
- PreNorm returns the result of add+residual, and PostNorm does not return the result.
- hasBias: If true, the bias tensor can be input. If false, the bias tensor is not input.
- Quantization
Supported only for PreNorm.
Parent topic: RmsNormOperation