[object Object]

Note: This API will be deprecated in later versions. Use the latest API instead.

[object Object][object Object]undefined[object Object]

• Description: FlashAttention operator in the full inference scenario.

• Formula:

  Self-attention constructs an attention model by leveraging the relationships within the input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n × d$ matrices.

  The computation formula for self-attention is generally defined as follows, where $Q$, $K$, and $V$ are key attribute elements of the input sample, obtained through spatial transformation and unified into a single feature space. "Attention" in the formula and operator name is an abbreviation for "self-attention."

  $$Attention(Q,K,V)=Score(Q,K)V$$

  In this operator, the [object Object] function is used, instead of the [object Object] function. The self-attention computation formula is as follows:

  $$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d}})V$$

  The product of $Q$ and $K^T$ represents the attention to the input $x$. To prevent this value from becoming excessively large, it is typically scaled by dividing by the square root of $d$, followed by row-wise softmax normalization. The result is then multiplied by $V$ to produce an $n × d$ matrix.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a deterministic implementation.

• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.

• Processing logic for a null input parameter: The operator checks whether [object Object] is a null pointer. If so, an error is reported. If [object Object] is not an empty tensor but [object Object] and [object Object] are empty tensors (that is, [object Object] is 0), [object Object] is filled with all zeros. If [object Object] is an empty tensor, the AscendCLNN framework will process it. For other input parameters which support the passing of [object Object] as described in the preceding parameter description, no processing is performed when they are null pointers.

• The data layout of [object Object], [object Object], and [object Object] can be interpreted from multiple dimensions. To be specific, [object Object] (Batch) indicates the size of an input sample batch, [object Object] (Seq-Length) indicates the length of the input sample sequence, [object Object] (Head-Size) indicates the size of the hidden layer, [object Object] (Head-Num) indicates the number of heads, and [object Object] (Head-Dim) indicates the minimum unit size of the hidden layer ([object Object] = [object Object]/[object Object]).

• Restrictions on [object Object], [object Object], and [object Object]:

  • Input shape restrictions:

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT:

      • The B axis must be less than or equal to 65536 (64K). If the input type is INT8 and the D axis is not 32-byte aligned, or the input type is FLOAT16 or BFLOAT16 and the D axis is not 16-byte aligned, the B axis can be up to 128.

      • The N axis must be less than or equal to 256.

      • The S axis must be less than or equal to 20971520 (20M). In some long sequence scenarios, if the computation load is too large, the PFA operator execution may time out (an AI Core error is reported, and [object Object] is [object Object]). In this case, S axis splitting is recommended. Note: The computation load is affected by parameters such as [object Object], [object Object], [object Object], and [object Object]. Larger values indicate larger computation loads. The following lists some typical scenarios with long sequences (that is, the product of [object Object], [object Object], [object Object], and [object Object] is large).

        [object Object]

      • The D axis must be less than or equal to 512. If [object Object] is [object Object] or [object Object], [object Object] must be less than 65535.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Constraints on the input of [object Object], [object Object], and [object Object] in the TND scenario:

      • [object Object] is less than or equal to 65536.
      • [object Object] is 8, 16, 32, 64, or 128, and [object Object], [object Object], and [object Object] are equal.
      • [object Object] and [object Object] are 192, and [object Object] is 128 or 192.
      • The data type is BFLOAT16.
      • The sparse mode can only be 0 without a mask or 3 with a mask.
      • When the sparse mode is 3, [object Object] must be less than [object Object] for each batch.

    • [object Object]Atlas inference accelerator cards[object Object]:

      • When inputLayout is BSH, the value of B axis must be less than or equal to 300. In other cases, the value of B axis must be less than or equal to 128.
      • The N axis must be less than or equal to 256.
      • The S axis must be less than or equal to 65535 (64K). [object Object] cannot be configured if [object Object] or [object Object] is not 128-aligned or [object Object] and [object Object] have different values.
      • The D axis must be less than or equal to 512.
  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can only be FLOAT16.

• Restrictions on [object Object]:

  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference accelerator cards[object Object]: The value can only be [object Object].

• Restrictions on [object Object]:

  • Input shape restrictions: Recommended shapes are [object Object], [object Object], [object Object], [object Object], and [object Object]. [object Object] is [object Object] in the shape of [object Object], and [object Object] is [object Object] in the shape of [object Object] and [object Object].
  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: BOOL, INT8, and UINT8 are supported.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can only be BOOL.
  • Other restrictions: In the scenario where [object Object] of [object Object] is not 32-byte aligned, it is recommended that it be padded to 32 bytes to improve the performance, filling excess positions with ones.

• Restrictions on the input of [object Object]:

  • Input value range restrictions: The valid sequence length of each batch in the input parameter should be less than or equal to the sequence length of the corresponding batch in [object Object].
  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: INT64 is supported.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can be INT64.

• Restrictions on the input of [object Object]:

  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: INT64 is supported.
    • [object Object]Atlas inference accelerator cards[object Object]: The value can only be [object Object].

• Restrictions on the input of [object Object]:

  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: The data type can be INT64.
    • [object Object]Atlas inference accelerator cards[object Object]: The value can only be [object Object] or [object Object].

• Restrictions on the input of [object Object]:

  • Input data type restrictions:
    • The input format can be [object Object], [object Object], [object Object], or [object Object]. When the input format is [object Object], the output format is [object Object]. If no specific value is required, [object Object] is recommended.

• Restrictions on the input of [object Object]:

  • Input attribute restrictions: [object Object] must be divisible by [object Object], and in BSND, BNSD, BNSD_BSND scenarios, it must match the N-axis value of [object Object]/[object Object] in the shape. Otherwise an error is reported.
  • Input data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: The data type can be INT64.
    • [object Object]Atlas inference accelerator cards[object Object]: The value can only be [object Object].

• Restrictions on the input of [object Object]:

  • Shape restrictions: When [object Object] is set to [object Object], the shape of the input [object Object] is BNSD and the output shape is BSND. In other cases, the shape of this input parameter must be the same as that of the input parameter [object Object].
  • Data type restrictions:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT: The data type can be FLOAT16 or BFLOAT16.
    • [object Object]Atlas inference accelerator cards[object Object]: The data type can only be FLOAT16.

[object Object]

The following example is for reference only. For details, see .