Documentation

NPULossScaleOptimizer Constructor

Description

Constructor of the NPULossScaleOptimizer class, which is used to enable loss scaling in mixed precision training when the overflow/underflow mode of floating-point computation is saturation mode. Loss scaling solves the underflow problem caused by the small float16 representation range. The NPULossScaleOptimizer class inherits the LossScaleOptimizer class and can call native APIs of the base class.
  • Atlas Training Series Product: The default overflow/underflow mode of floating-point computation is saturation mode, and only the saturation mode is supported. This means when an overflow/underflow occurs during computation, the computation result is saturated to a floating-point extreme value (±MAX).

Prototype

def __init__(self, opt, loss_scale_manager, is_distributed=False)

Options

Option

Input/Output

Description

opt

Input

Single-server training optimizer for gradient calculation and weight update.

loss_scale_manager

Input

Loss scaling update mode, including static update and dynamic update.

  • Before creating NPULossScaleOptimizer, you can instantiate a FixedLossScaleManager class to set the loss scaling with a static value. For details about the constructor of the FixedLossScaleManager class, see FixedLossScaleManager Constructor.
  • Before creating NPULossScaleOptimizer, you can instantiate an ExponentialUpdateLossScaleManager class to dynamically configure loss scaling. For details about the constructor of the ExponentialUpdateLossScaleManager class, see ExponentialUpdateLossScaleManager Constructor.

is_distributed

Input

Used to support the loss scaling function in the distributed training scenario.

  • True: Set it to True for distributed training.
  • False

Returns

An object of the NPULossScaleOptimizer class

Example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from npu_bridge.npu_init import *

if FLAGS.use_fp16 and (FLAGS.npu_bert_loss_scale not in [None, -1]):
  opt_tmp = opt
  if FLAGS.npu_bert_loss_scale == 0:
    loss_scale_manager = ExponentialUpdateLossScaleManager(init_loss_scale=2**32, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, decr_ratio=0.5)
  elif FLAGS.npu_bert_loss_scale >= 1:
    loss_scale_manager = FixedLossScaleManager(loss_scale=FLAGS.npu_bert_loss_scale)
  else:
    raise ValueError("Invalid loss scale: %d" % FLAGS.npu_bert_loss_scale)
  if ops_adapter.size() > 1:
    opt = NPULossScaleOptimizer(opt_tmp, loss_scale_manager, is_distributed=True)
  else:
    opt = NPULossScaleOptimizer(opt_tmp, loss_scale_manager)