[object Object][object Object][object Object]undefined[object Object]

• API function: Uses the FlashAttention algorithm to perform self-attention computation in training scenarios. [object Object] is added as an optional parameter.

  • Ascend 950PR/Ascend 950DT: The sinkInOptional parameter is not supported currently.

• Formula:

  The forward propagation formula for attention is as follows:

  $$=Dropout(Softmax(Mask(\frac{QK^T+pse}{\sqrt{d}}),atten\_mask),keep\_prob)V$$

  For convenience, the formula can be represented using variables $S$ and $P$:

  $$=Mask(\frac{QK^T+pse}{\sqrt{d}}),atten\_mask$$ $$=Dropout(Softmax(S),keep\_prob)$$ $$=PV$$

  Then the backward propagation formula for attention is as follows:

  $$V=P^TdY$$ $$Q=\frac{((dS)*K)}{\sqrt{d}}$$ $$K=\frac{((dS)^T*Q)}{\sqrt{d}}$$

  The following calculation logic applies after sink is added, primarily modifying the calculation parts related to softmax_max and softmax_sum:

  $$S = Q @ K^{T}$$ $$m = max(sink, max(S))$$

$$Attention = \frac{e^{S - m} @ V}{\sum e^{S-m} + e^{sink - m}}$$ $$dSink = reduce(-P * dP * SimpleSoftmax(sink, x\_max, x\_sum))$$ [object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a non-deterministic implementation. You can call [object Object] to enable deterministic computing.

• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.

• B (batch size) of the input [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be the same.

• The [object Object] of the input [object Object], [object Object], [object Object], and [object Object] must be the same.

• [object Object] of the input [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] must be TND.

• The input data types of [object Object], [object Object], [object Object], and [object Object] must be the same.

• The shapes of the input [object Object] and [object Object] must be the same. If the D values of [object Object], [object Object], and [object Object] are the same, the shapes of [object Object] and [object Object] must be the same. Note: In the current version, the D values of [object Object], [object Object], and [object Object] must be the same.

• N of the input [object Object] can be different from N of the [object Object] or [object Object], but they must be proportional. That is, Nq/Nkv must be a non-zero integer and the value of Nq ranges from 1 to 256.

• The following uses the [object Object] TND as an example to describe the constraints on the shape:

  • T(B*S): The value ranges from 1 to 1M.
  • B: The value ranges from 1 to 20000. When prefixOptional is carried, B supports a maximum of 1K.
  • N: The value ranges from 1 to 256.
  • S: The value ranges from 1 to 1M.
  • D: The value ranges from 1 to 768.
  • KeepProb: The value range is (0, 1].

• The data format of [object Object], [object Object], and [object Object] can only be TND. T indicates the data closely arranged on the B and S axes (SeqLenQ and SeqLenKV of each batch). B (Batch) indicates the batch size of the input sample, and S (Seq-Length) indicates the length of the input sample sequence. H (Head-Size) indicates the size of the hidden layer, N (Head-Num) indicates the number of heads. D (Head-Dim) indicates the minimum unit size of the hidden layer (D = H/N).

• pseShiftOptional: If Sq is greater than 1024, the length of Sq in each batch is the same as that of Skv, and the sparseMode is 0, 2, or 3, the alibi positional encoding compression can be enabled. In this case, only the last 1024 rows of the original PSE need to be input to optimize the memory. That is, alibi_compress = ori_pse[:, :, -1024:, :]. The details are as follows:

  • If the parameters of each batch are different, the shape is BNHSkv (H=1024).
  • When each batch is the same, the shape is 1NHSkv (H=1024).
  • If [object Object] is 2 or 3, the data type must be FLOAT32, and the supported shapes are [B,N] and [N].
  • If this parameter is not enabled, pass a null pointer to [object Object] and 1 to [object Object].

• Meanings of [object Object] values:

  [object Object]undefined

• The constraints for [object Object] are as follows:

  • If the shape values of all [object Object] are the same and less than 2048, you are advised to use the default mode to reduce memory usage.
  • When this parameter is set to 1, 2, or 3, the configured preTokens and nextTokens do not take effect.
  • When the value is set to 0 or 4, ensure that the ranges of [object Object], [object Object], and [object Object] are consistent.
  • If no specific value is required, you are advised to set it to 0.
  • For details about the sparse modes, see .
  • When this parameter is set to 7, the optional input pseShiftOptional is not supported.
  • When this parameter is set to 8, the optional input pseShiftOptional is supported when the q and kv of each sequence have the same length. In this case, the PSE is generated globally. The q direction can be used for external splitting. The q and kv of each sequence must have the same length before external splitting. Then, actualSeqQLenOptional[0] - actualSeqKvLenOptional[0] + qStartIdxOptional - kvStartIdxOptional == 0 (experimental function).

• For details about different data formats, see .

• In some scenarios, if the computation load is too large, the operator execution may time out (AI Core error, errorStr: timeout or trap error). In this case, you are advised to perform axis splitting. Note: The computation load is affected by parameters such as B, S, N, and D. Larger values indicate larger computation loads.

• The [object Object] sparse computing supports only compression scenarios ([object Object]). When Sq > Skv, the value range of N of [object Object] is [0, Skv]. When Sq ≤ Skv, the value range of N of [object Object] is [Skv – Sq, Skv].

• The [object Object] input supports the S length of 0 in a batch. In this case, the [object Object] input is not supported.

• Constraints on the [object Object] and [object Object] parameters: The input format is fixed at [B, N, S, 8], except TND format, which is [T, N, 8]. Note: T = B x S.

• The value of [object Object] must be the same as the value of N in [object Object].

• When the data layout of [object Object] and [object Object] is TND, [object Object] must be set to [object Object].

• The sinkInOptional dimension is 1, and its length must be the same as that of headNum in the query.

[object Object]

The following example is for reference only. For details, see .