FixedLossScaleManager Constructor

Description

Constructor of the FixedLossScaleManager class, which is used to define the static LossScale parameter during training when the overflow/underflow mode of floating-point computation is saturation mode.

Atlas Training Series Product: The default overflow/underflow mode of floating-point computation is saturation mode, and only the saturation mode is supported. This means when an overflow/underflow occurs during computation, the computation result is saturated to a floating-point extreme value (±MAX).

Prototype

def __init__(self, loss_scale, enable_overflow_check=True)

Options

Option	Input/Output	Description
loss_scale	Input	Loss scale value. The value is of the float type and cannot be less than 1. If the value of loss scale is too small, model convergence may be affected. If the value of loss scale is too large, overflow may occur during training. The value can be the same as that of GPU.
enable_overflow_check	Input	Overflow detection enable during parameter update. True (default): enabled. If overflow is detected in an iteration, parameters of that iteration are not updated. False: disabled. Parameters are always updated regardless of overflow.

Option

Input/Output

Description

loss_scale

Input

Loss scale value. The value is of the float type and cannot be less than 1.

If the value of loss scale is too small, model convergence may be affected. If the value of loss scale is too large, overflow may occur during training. The value can be the same as that of GPU.

enable_overflow_check

Input

Overflow detection enable during parameter update.

True (default): enabled. If overflow is detected in an iteration, parameters of that iteration are not updated.
False: disabled. Parameters are always updated regardless of overflow.

Returns

An object of the FixedLossScaleManager class

Parent topic: npu_bridge.estimator.npu.npu_loss_scale_manager