FastSoftMaxGradOperation
Description
Training operator, which is used to calculate the reverse result of FastSoftMax. The data arrangement structure is the same as that of FastSoftMax.
Application Scenarios
The FastSoftMaxGrad operator is used to calculate the result of FastSoftMax. The data arrangement structure is the same as that of FastSoftMax.
Definition
struct FastSoftMaxGradParam {
int32_t headNum = 0;
std::vector<int32_t> qSeqLen;
uint8_t rsv[8] = {0};
};
Parameters
|
Member |
Type |
Default Value |
Description |
|---|---|---|---|
|
headNum |
int32_t |
0 |
Number of attention heads. The value must be greater than 0. |
|
qSeqLen |
std::vector<int32_t> |
- |
Actual input length of each batch. The number of elements is the same as the batch size. The maximum value is 32. |
|
rsv[8] |
uint8_t |
{0} |
Reserved |
Input
|
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
|
yInput |
[nSquareTokens] |
float16 |
ND |
Input tensor. The forward computation result of the SoftMax operator is the one-dimensional result obtained by concatenating batch (headNum, qSeqLen[i], qSeqLen[i]) matrices based on the ND layout. |
|
yGrad |
[nSquareTokens] |
float16 |
ND |
Gradient data transferred by the next operator. The layout is the same as that of yInput. |
Output
|
Parameter |
Dimension |
Data Type |
Format |
Description |
|---|---|---|---|---|
|
output |
[nSquareTokens] |
float16 |
ND |
The layout is the same as that of yInput. |
Restrictions
- The length of the qSeqLen array cannot exceed 32, and each element must be greater than 0.
- Currently, only the
Atlas A2 inference products is supported. - The dimension size nSquareTokens of the input tensors (yInput and yGrad) and output tensor (outTensor) is related to headNum and qSeqLen in the parameters.
: indicates the kth element in qSeqLen.
Functions
When the parameter list contains
:
The shape size of the corresponding yInput is as follows:
.
For example,
In this case, the shape size of yInput is
The shape of the input tensors yInput and yGrad, and output tensor is [5528144].