Mixed Precision Training on the NPU

Objective

The model is successfully ported to the NPU before accuracy debugging. Distributed training (if involved) is enabled.

Especially, mixed precision training is enabled during model porting.

Procedure

There are two methods to enable mixed precision training on the GPU:

Manual mixed precision training on the GPU: All operator data types have been defined in the model.
Automatic mixed precision training on the GPU: Operator data types are defined using TensorFlow or other third-party APIs (such as apex APIs).

Note that you should enable only one of the preceding methods to avoid unexpected problems invited by frequent graph modification. In addition, the same method should be used in case of porting to the NPU and the NPU accuracy configuration is as follows:

For the Atlas A3 training products/Atlas A3 inference products, use the precision_mode_v2 parameter. The value is origin.

For the Atlas A2 training products/Atlas A2 inference products, use the precision_mode_v2 parameter. The value is origin.

For the Atlas training products, use the precision_mode parameter. The value is allow_fp32_to_fp16.

Parent topic: Ported Script Check