[object Object][object Object][object Object]undefined[object Object]

• Uses the FlashAttention algorithm to perform self-attention computation in training scenarios. Compared with , this API supports multiple query/key inputs, that is, [object Object], [object Object], [object Object], and [object Object] are used as inputs. In non-multi-input scenarios, use or other APIs.

• Formula

  The forward computation formula for attention is as follows:

  $$attention\_out=Dropout(Softmax(Mask(scale*(query*key^T + queryRope*keyRope^T) + pse),atten\_mask),keep\_prob)*value$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation
  • [object Object] defaults to a deterministic implementation.
• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.
• B (batch size) of the input [object Object], [object Object], [object Object], [object Object], and [object Object] must be the same.
• D (Head-Dim) of the input [object Object], [object Object], and [object Object] must satisfy (qD == kD && kD >= vD). D must be an integer multiple of 8.
• D (Head-Dim) of the input [object Object] and [object Object] must satisfy (qRopeD == kRopeD). D must be an integer multiple of 8 and less than or equal to the D of [object Object], [object Object], and [object Object].
• The [object Object] of the input [object Object], [object Object], and [object Object] must be TND.
• Constraints on the data shape:
  • T: The value ranges from 1 to 1M.
  • N: The value ranges from 1 to 256.
  • D: The value ranges from 1 to 768.
  • The data shape must be TND.
  • [object Object] must be 1.
• The data format of [object Object], [object Object], and [object Object] can only be TND. T indicates the data closely arranged on the B and S axes (SeqLenQ and SeqLenKV of each batch). B (Batch) indicates the batch size of the input sample, and S (Seq-Length) indicates the length of the input sample sequence. H (Head-Size) indicates the size of the hidden layer. N (Head-Num) indicates the number of heads. D (Head-Dim) indicates the minimum unit size of the hidden layer (D = H/N).
• [object Object]: 0 and 1 are reserved, and 2 indicates that invalid row calculation is enabled. This function is used to prevent precision loss caused by the mask of the entire row during calculation. However, this configuration deteriorates the performance. If the operator can determine that invalid rows exist, the invalid row computation is automatically enabled, such as in scenarios where [object Object] is set to 3 and Sq is greater than Skv.
• Restrictions on [object Object]
  • When the shapes of all [object Object] are less than 2048 and are the same, the default mode is recommended to reduce memory usage.
  • When this parameter is set to 1, 2, or 3, the configured preTokens and nextTokens do not take effect.
  • When the value is set to 0 or 4, ensure that the ranges of [object Object], [object Object], and [object Object] are consistent.
  • If no specific value is required, 0 is recommended.
  • For details about the sparse modes, see .
  • When the value is set to 3, computation on invalid rows is not supported, and Sq <= Skv must be satisfied for each batch.
  • When the value is set to 7, [object Object] is not supported.
  • When the value is set to 8, [object Object] is supported when the q and kv of each sequence have the same length. PSE generation is performed globally. The q direction can be used for external splitting. The q and kv of each sequence must have the same length before external splitting. Then, actualSeqQLenOptional[0] - actualSeqKvLenOptional[0] + qStartIdxOptional - kvStartIdxOptional == 0 (experimental function).
• In some scenarios, if the computation load is too large, the operator execution may time out (an AI Core error is reported, and [object Object] is [object Object]). In this case, you are advised to perform axis splitting. Note: The computation load is affected by parameters such as B, S, N, and D. Larger values indicate larger computation loads.
• In the band scenario, the values of [object Object] and [object Object] must overlap.
• The [object Object] input supports the S length of 0 in a batch. In this case, the [object Object] input is not supported. The length of [object Object] ranges from 1 to 2K. When [object Object] is present, its maximum length is 1K.
• The [object Object] input does not support padding. That is, [object Object] cannot contain a row of all 1s.
• The S length of a batch in [object Object] can be 0. If the S length is 0, the [object Object] input is not supported. If the actual S length is [2,2,0,2,2], the value of [object Object] is [2,4,4,6,8].
• pseType can only be 1.
• [object Object] must be null.
• [object Object] must be null.
• [object Object] cannot be null.

[object Object]

The following example is for reference only. For details, see .