Mixed Precision Training
Objective
If the benchmark model is trained in float32 on the GPU or CPU, the prediction does not change in mixed precision training (using float16 on the GPU).
If the benchmark model is trained in mixed precision mode (in float16 on the GPU), the prediction does not change in float32 training on the GPU or CPU.
If the result of training in high-precision mode fails to be obtained due to memory or hardware limitations, ensure that a constant model accuracy is achieved.
Principle
As AI processor (or NPU) hardware architecture supports only mixed precision training, the user model must be trained on the GPU with mixed precision to obtain a converged benchmark model. If the user model is not validated to reach convergence after mixed precision training on the GPU, the model may fail to converge after being ported to the NPU.
The model is not qualified for a benchmark if the comparison between mixed-precision training and high-precision training fails to meet the accuracy requirements, as this means that the fp16 data type in mixed-precision training causes significant accuracy differences in this training scenario, posing accuracy risks for both GPU and NPU mixed-precision training. In this case, the model structure needs to be adjusted to ensure that mixed-precision training can converge.
Procedure
- Check that training with mixed precision is enabled for the benchmark model.
- Check that dynamic loss scaling is enabled.
You can also enable static loss scaling, but it is not recommended, as you need to adjust the loss scale value on the NPU to avoid frequent overflow or underflow when using the loss scale value set on the GPU.
- Check the proportion of floating-point exceptions reported during mixed precision training. A percentage less than 0.5% (0.1% for a large global batch size) is recommended.
- Modify the initial value and scaling factor of the loss scale to minimize the number of floating-point exceptions.
- Perform training more than three times and check that the validation accuracy is constant. Adjust the hyperparameters and model structure, and repeat the preceding steps until all the preceding conditions are met.