[object Object]

[object Object][object Object]undefined[object Object]

• Description: This operator is a part of the cross entropy computation module in the vocabulary parallelism scenario. It solves the video memory and computing efficiency problems in the case of ultra-large vocabulary. This part involves the gradient computation implementation, which is used to compute the gradients of leaf nodes [object Object] and [object Object]. The outputs of [object Object] and [object Object], along with the global communication result related to [object Object], need to be obtained as the input of this API.
• Formula:

   High-performance mode, where the value of softmaxOptional is not nullptr:

$$\text{softmax} \in \mathbb{R}^{BT \times V}$$ $$\text{arange\_1d} = [0, 1, \dots, BT-1] \in \mathbb{N}^{BT}$$ $$\text{softmax\_update} = \mathbf{1} - \text{target\_mask}.view(-1) \in \mathbb{R}^{BT}$$ $$\text{softmax}[\text{arange\_1d}, \text{masked\_target}] \leftarrow \text{softmax}[\text{arange\_1d}, \text{masked\_target}] - \text{softmax\_update}$$ $$\text{softmax} \leftarrow \text{softmax} \odot \text{grad}.unsqueeze(-1) \in \mathbb{R}^{BT \times V}$$ $$\text{grad\_input} = \text{softmax} \cdot \text{weight}^T \in \mathbb{R}^{BT \times H}$$ $$\text{grad\_weight} = \text{softmax}^T \cdot \text{input} \in \mathbb{R}^{V \times H}$$ [object Object]$$\text{vocab\_parallel\_logits} = \text{input} \cdot \text{weight}^T \quad \in \mathbb{R}^{BT \times V}$$ $$\text{logits\_sub} = \text{vocab\_parallel\_logits} - \text{logits\_max}.unsqueeze(-1) \quad \in \mathbb{R}^{BT \times V}$$ $$\text{exp\_logits} = \exp(\text{logits\_sub}) \quad \in \mathbb{R}^{BT \times V}$$ $$\text{exp\_logits} \gets \frac{\text{exp\_logits}}{\text{sum\_exp\_logits}.unsqueeze(-1)} \quad \in \mathbb{R}^{BT \times V}$$ $$\text{grad\_logits} = \text{exp\_logits} \quad \in \mathbb{R}^{BT \times V}$$ $$\text{grad\_2d} = \text{grad\_logits}.view(-1, \text{partition\_vocab\_size}) \quad \in \mathbb{R}^{BT \times V}$$ $$\text{arange\_1d} = [0, 1, \dots, BT-1] \quad \in \mathbb{N}^{BT}$$ $$\text{softmax\_update} = 1 - \text{target\_mask}.view(-1) \quad \in \mathbb{R}^{BT}$$ $$\text{grad\_2d}[\text{arange\_1d}, \text{masked\_target\_1d}] \gets \text{grad\_2d}[\text{arange\_1d}, \text{masked\_target\_1d}] - \text{softmax\_update}$$ $$\text{grad\_logits} \gets \text{grad\_logits} \odot \text{grad}.unsqueeze(-1) \quad \in \mathbb{R}^{BT \times V}$$ $$\text{grad\_input} = \text{grad\_logits} \cdot \text{weight} \quad \in \mathbb{R}^{BT \times H}$$ $$\text{grad\_weight} = \text{grad\_logits}^T \cdot \text{input} \quad \in \mathbb{R}^{V \times H}$$ [object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic description:
  • The aclnnFusedLinearCrossEntropyLossGrad is implemented in a deterministic manner by default.

[object Object]

The following example is for reference only. For details, see .