FixedLossScaleManager Constructor

Applicability

Product	Supported (√/x)
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	☓
Atlas inference product	☓
Atlas training product	√

Description

Constructor of the FixedLossScaleManager class, which is used to define the static LossScale parameter during training when the overflow/underflow mode of floating-point computation is saturation mode.

For the Atlas 350 Accelerator Card, the overflow/underflow mode of floating-point computation can be saturation or Inf/NaN. Retain the default Inf/NaN mode. The saturation mode is used only for compatibility with earlier versions and will not evolve in the future. In addition, the computing accuracy in this mode may be unreliable.
For the Atlas A3 training product/Atlas A3 inference product, the overflow/underflow mode of floating-point computation can be saturation or Inf/NaN. Retain the default Inf/NaN mode. The saturation mode is used only for compatibility with earlier versions and will not evolve in the future. In addition, the computing accuracy in this mode may be unreliable.
For the Atlas A2 training product/Atlas A2 inference product, the overflow/underflow mode of floating-point computation can be saturation or Inf/NaN. Retain the default Inf/NaN mode. The saturation mode is used only for compatibility with earlier versions and will not evolve in the future. In addition, the computing accuracy in this mode may be unreliable.
For the Atlas training product, the default overflow/underflow mode of floating-point computation is saturation, and only the saturation mode is supported. This means when an overflow/underflow occurs during computation, the computation result is saturated to a floating-point extreme value (±MAX).

Prototype

class FixedLossScaleManager(lsm_lib.FixedLossScaleManager):
    def __init__(self, loss_scale, enable_overflow_check=True)

Parameters

Parameter	Input/Output	Description
loss_scale	Input	Loss scale value. The value is of the float type and cannot be less than 1. If the value of loss scale is too small, model convergence may be affected. If the value of loss scale is too large, overflow may occur during training. The value can be the same as that of GPU.
enable_overflow_check	Input	Overflow detection enable during parameter update. True (default): enabled. If overflow is detected in an iteration, parameters of that iteration are not updated. False: disabled. Parameters are always updated regardless of overflow.

Parameter

Input/Output

Description

loss_scale

Input

Loss scale value. The value is of the float type and cannot be less than 1.

If the value of loss scale is too small, model convergence may be affected. If the value of loss scale is too large, overflow may occur during training. The value can be the same as that of GPU.

enable_overflow_check

Input

Overflow detection enable during parameter update.

True (default): enabled. If overflow is detected in an iteration, parameters of that iteration are not updated.
False: disabled. Parameters are always updated regardless of overflow.

Returns

An object of the FixedLossScaleManager class

Parent topic: npu_bridge.estimator.npu.npu_loss_scale_manager