[object Object]

Note: This API will be deprecated in later versions. Use the latest API instead.

[object Object][object Object]undefined[object Object]

• Description: For an auto-regressive language model, the length of the inference input increases as new words are generated. Incremental inference is implemented based on full inference. The S axis of [object Object] is fixed at [object Object]. The [object Object] and [object Object] are their respective content in the KV cache plus the state information generated by inference. The actual length of the S axis corresponding to each batch may be different, and the input data is the fixed-length data after padding.

  Compared with the FlashAttention operator () in the full inference scenario, this operator has a different inference process but similar accuracy (in the incremental inference scenario).

  Note:

  The KV cache is a common technology for optimizing the inference performance of foundation models. During sampling, the transformer model uses the given prompt/context as the initial input for inference (parallel processing is supported), and then generates additional tokens one by one to improve the generated sequence (reflecting the auto-regressive property of the model). The transformer performs the self-attention operation during sampling. Therefore, KV vectors need to be extracted for each item (regardless of the prompt/context or generated token) in the current sequence. These vectors are stored in a matrix called KV cache.

• Formula:

  Self-attention constructs an attention model by leveraging the relationships within input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n$d matrices.

  The calculation formula for self-attention is generally defined as follows, where $Q$, $K$, and $V$ are key attribute elements of the input sample, obtained through spatial transformation and unified into a single feature space.

  $$Attention(Q,K,V)=Score(Q,K)V$$

  The [object Object] function in this operator employs the [object Object] function. The self-attention calculation formula is as follows:

  $$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d}})V$$

  The product of $Q$ and $K^T$ represents the attention to the input $x$. To prevent this value from becoming too large, the product is usually divided by the square root of $d$ for scaling, normalized by softmax for each row, and multiplied by $V$ to result in an $n*d$ matrix.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.
• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and Ascend 950PR/Ascend 950DT:
  • The B axis can be less than or equal to [object Object], the N axis can be less than or equal to [object Object], and the D axis can be less than or equal to [object Object].
  • The data type of [object Object] can be FLOAT16 or BFLOAT16. The data types of [object Object], [object Object], and [object Object] can be FLOAT16 or BFLOAT16.
  • The data type of [object Object] can be INT64.
• [object Object]Atlas inference accelerator cards[object Object]:
  • The B axis can be less than or equal to [object Object], the N axis can be less than or equal to [object Object], and the D axis can be less than or equal to [object Object].
  • The S axis of [object Object] and [object Object] is less than or equal to [object Object].
  • The data types of [object Object], [object Object], [object Object], and [object Object] can only be [object Object].
  • [object Object] supports only the value [object Object].
• In non-contiguous scenarios, the number of tensors in the tensor list of parameters [object Object] and [object Object] is equal to [object Object] in [object Object] (due to tensor list restrictions, [object Object] must be less than or equal to [object Object] in non-contiguous scenarios). The shapes (except for that of [object Object]) must be identical, and the batch size can only be [object Object].
• The values of [object Object] and [object Object] are the same in [object Object]. The values of [object Object] and [object Object] are the same in [object Object] and [object Object]. The value of [object Object] is a multiple of that of [object Object].
• The S axis of [object Object] can only be [object Object].
• When the data type of [object Object] is INT8 or UINT8, the value in the tensor must be [object Object] or [object Object].

[object Object]

The following example is for reference only. For details, see .