[object Object][object Object][object Object]undefined[object Object]

• Description: Computes the attention backpropagation output in training scenarios, which is the backpropagation of . The query, key, and value parameters of this API support multiple sequences with the same or different lengths.

  • This API combines the and APIs and adjusts the Dropout function.
    • Ascend 950PR/Ascend 950DT: When keepProb is less than 1.0, if there is no external DropoutMask, the new parameter is used to generate the DropoutMask. If there is an external DropoutMask, the external DropoutMask is used.

• Formulas:

  • When pseType is set to 1, the calculation formula is the same as that of .
  • When pseType is set to other values, the formula is as follows:
  $$Y=Dropout(Softmax(Mask(\frac{QK^T}{\sqrt{d}}+pse),atten\_mask),keep\_prob)V$$

  For convenience, the formula can be represented using variables $S$ and $P$:

  $$S=Mask(\frac{QK^T}{\sqrt{d}}+pse),atten\_mask$$ $$P=Dropout(Softmax(S),keep\_prob)$$ $$Y=PV$$

  Then the backward propagation formula for attention is as follows:

  $$dV=P^TdY$$ $$dQ=\frac{((dS)*K)}{\sqrt{d}}$$ $$dK=\frac{((dS)^T*Q)}{\sqrt{d}}$$

  NOTE The data formats of query, keyIn, and value can be interpreted from multiple dimensions. T (Total S Length) indicates the total length of S corresponding to all batches, B (Batch) indicates the batch size of the input sample, S (Seq-Length) indicates the length of the input sample sequence, H (Head-Size) indicates the size of the hidden layer, N (Head-Num) indicates the number of heads, and d (Head-Dim) indicates the minimum unit size of the hidden layer. d = H/N.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to non-deterministic implementation. You can call [object Object] to enable deterministic computation.

• The restrictions on the inputs query, key, value, and dy are as follows:

  • B: The batch sizes must be equal.
  • [object Object] must be consistent.
  • D: The head dimension (D) of the query and key must be the same, and that of the value and dy must be the same. In addition, the D of the query and key must be greater than or equal to that of the value and dy.

• N of the input [object Object] or [object Object] can be different from N of the [object Object] or [object Object], but they must be proportional. That is, Nq/Nkv must be a non-zero integer. The value of Nq ranges from 1 to 256.

• The following uses the [object Object] TND as an example to describe the constraints on the shape:

  • B: The value ranges from 1 to 2K. When [object Object] is passed, B supports a maximum of 1K.
  • N: The value ranges from 1 to 256.
  • S: The value ranges from 1 to 1M.
  • D: The value ranges from 1 to 768.
  • KeepProb: The value range is (0, 1].

• In some scenarios, if the computation load is too large, the operator execution may time out (an AI Core error is reported, and [object Object] is [object Object]). In this case, you are advised to perform axis splitting. Note: The computation load is affected by parameters such as B, S, N, and D. Larger values indicate larger computation loads.

• The prefixOptional sparse computing supports only compression scenarios. sparseModeOptional = 6. When Sq > Skv, the value range of N of prefix is [0, Skv]. When Sq ≤ Skv, the value range of N of prefix is [Skv – Sq, Skv]. When sparseModeOptional is set to 5 or prefixOptional is not passed, full computation is performed. When sparseModeOptional is set to 6, prefixOptional must be passed.

• When sparseMode is set to 7, the optional input pseShiftOptional is not supported.

• When sparseMode is set to 8 and the length of q and kv in each sequence is the same, the optional input pseShiftOptional is supported. The global PSE is generated. The q direction can be used for external splitting. The q and kv of each sequence must have the same length before external splitting. Then, actualSeqQLenOptional[0] - actualSeqKvLenOptional[0] + qStartIdxOptional - kvStartIdxOptional == 0 (experimental function).

• The [object Object] input supports the S length of 0 in a batch. In this case, the [object Object] input is not supported.

• The restrictions on the softmaxMax and softmaxSum parameters are as follows: The input format is fixed to [B, N, S, 8], except for the TND input format, which is [N, T, 8]. Note that T = B x S.

• The value of [object Object] must be the same as the value of N in [object Object].

• Ascend 950PR/Ascend 950DT:

  • seedOptional and offsetOptional take effect only when keepProbOptional is less than 1.0. Otherwise, they do not take effect.
  • When keepProbOptional is less than 1.0, if dropMaskOptional is not nullptr, the input dropMask is used. Otherwise, the dropMask generated by seed and offset is used.

• In TND format, some batches at the end are not involved in the calculation. In this case, you can pass zeros of the corresponding number to the end of actual_seq_q_len and actual_seq_kv_len. Assume that the actual S length is [2, 3, 4, 5, 6]. If the last two batches are not involved in the calculation, the input actual_seq_q_len is [2, 3, 4, 0, 0]. In this case, if prefixOptional needs to be passed, the same number of zeros also needs to be passed to the end, for example, [1, 1, 1, 0, 0].

[object Object]

The following example is for reference only. For details, see .