[object Object]

[object Object][object Object]undefined[object Object]

• Description: Implements the FusedEmaAdam fusion optimizer.
• Formula:$$(correction_{\beta_1},correction_{\beta_2},)=\begin{cases} (1,1),&biasCorrection=False\\ (1-\beta_1^{step},1-\beta_2^{step}),&biasCorrection=True \end{cases}$$ $$grad=\begin{cases} grad+weightDecay*var,&mode=0\\ grad,&mode=1 \end{cases}$$ $$m_{out}=\beta_1*m+(1-\beta_1)*grad$$ $$v_{out}=\beta_2*v+(1-\beta_2)*grad^2$$ $$m_{next}=m_{out}/correction_{\beta_1}$$ $$v_{next}=v_{out}/correction_{\beta_2}$$ $$denom=\sqrt{v_{next}}+eps$$ $$update=\begin{cases} m_{next}/denom,&mode=0\\ m_{next}/denom+weightDecay*var,&mode=1 \end{cases}$$ $$var_{out}=var-lr*update$$ $$s_{out}=emaDecay*s+(1-emaDecay)*var_{out}$$

[object Object]

Each operator has calls. First, aclnnApplyFusedEmaAdamGetWorkspaceSize is called to obtain the input, the workspace size required for computation, and the executor that contains the operator computation process. Then, aclnnApplyFusedEmaAdam is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • grad (aclTensor*, compute input): gradient of the parameter to be updated, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The can be ND.
  • varRef (aclTensor*, compute input/output): parameter to be updated, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The shape and data type must be the same as those of grad. The can be ND.
  • mRef (aclTensor*, compute input/output): first-order momentum of the parameter to be updated, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The shape and data type must be the same as those of grad. The can be ND.
  • vRef (aclTensor*, compute input/output): second-order momentum corresponding to the parameter to be updated, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The shape and data type must be the same as those of grad. The can be ND.
  • sRef (aclTensor*, compute input/output): EMA weight corresponding to the parameter to be updated, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The shape and data type must be the same as those of grad. The can be ND.
  • step (aclTensor*, compute input): number of updates of the optimizer, corresponding to [object Object] in the formula. It is an aclTensor on the device. The data type can be INT64. The can be ND.
  • lr (double, compute input): learning rate, corresponding to [object Object] in the formula.
  • emaDecay (double, compute input): decay rate of exponential moving average (EMA), corresponding to [object Object] in the formula.
  • beta1 (double, compute input): coefficient for calculating the first-order momentum, corresponding to $\beta_1$ in the formula.
  • beta2 (double, compute input): coefficient for calculating the second-order momentum, corresponding to $\beta_2$ in the formula.
  • eps (double, compute input): added to the denominator for numerical stability, corresponding to [object Object] in the formula.
  • mode (int64_t, compute input): controls whether to apply L2 regularization or weight decay, corresponding to [object Object] in the formula. The value 1 indicates adamw, and the value 0 indicates L2.
  • biasCorrection (bool, compute input): controls whether to correct the bias, corresponding to [object Object] in the formula. The value true indicates that the correction is performed, and the value false indicates that the correction is not performed.
  • weightDecay (double, compute input): weight decay, corresponding to [object Object] in the formula.
  • workspaceSize (uint64_t*, output): size of the workspace to be allocated on the device.
  • executor (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns:

  aclnnStatus status code. For details, see .

[object Object]

• Parameters:

  • workspace (void*, input): address of the workspace to be allocated on the device.
  • workspaceSize (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling the first-phase API aclnnApplyFusedEmaAdamGetWorkspaceSize.
  • executor (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • stream (aclrtStream, input): stream for executing the task.

• Returns:

  aclnnStatus status code. For details, see .

[object Object]

The data types and shapes of the input grad, var, m, v, and s must be the same.

• Deterministic compute:
  • aclnnApplyFusedEmaAdam defaults to a deterministic implementation.

[object Object]

The following example is for reference only. For details, see .