[object Object]

[object Object][object Object]undefined[object Object]

• Adapts to the [object Object] operator in the decode ([object Object]) and prefill ([object Object]) inference scenarios.

  Compared with FusedInferAttentionScoreV3, this API adds the dequantScaleQueryOptional, learnableSinkOptional, and queryQuantMode parameters, and the fullmask capability of alibi.

  Note: KV cache specific to the decode scenario: KV cache is a common technology for optimizing the inference performance of foundation models. During sampling, the transformer model uses the given prompt/context as the initial input for inference (parallel processing supported), and then generates additional tokens one by one to improve the generated sequence (reflecting the auto-regressive property of the model). The transformer performs the self-attention operation during sampling. Therefore, KV vectors need to be extracted for each item (regardless of the prompt/context or generated token) in the current sequence. These vectors are stored in a matrix called KV cache.

• Formula:

  Self-attention constructs an attention model by leveraging the relationships within the input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n × d$ matrices.

  The computation formula for self-attention is generally defined as follows, where $Q$, $K$, and $V$ are key attribute elements of the input sample, obtained through spatial transformation and unified into a single feature space. "Attention" in the formula and operator name is an abbreviation for "self-attention."

  $$Attention(Q,K,V)=Score(Q,K)V$$

  In this operator, the [object Object] function is used, instead of the [object Object] function. The self-attention computation formula is as follows:

  $$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d}} + FullMask)V$$

  The product of Q and K^T represents the attention of the input x. To avoid the value being too large, the value is usually scaled by dividing the square root of d, and the fullmask of alibi is added. Then, softmax normalization is performed on each row, and the result is multiplied by V to obtain an n x d matrix.

  Note:

  [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• Common constraints
  • Processing when the input parameter is empty:
    • An empty tensor indicates that the shapeSize of the required input and output is 0. In the empty tensor scenario, if attentionOut is empty, the empty tensor is returned. Otherwise, all 0s are returned. If lse is empty, the empty tensor is returned. If lse is not empty, all inf values are returned. When the tensor is not empty, the input is intercepted normally.
    • The shapeSize of all tensors in query and attentionOut is 0, which means that the tensor is empty.
    • The shapeSize of all tensors in query and attentionOut is not 0. If lse is not empty and the shapeSize of all tensors in key and value is 0, the tensor is empty.
    • If both attentionOut and lse are empty, the tensor is empty.
    • If the tensor is empty, the verification process is skipped. Otherwise, the normal verification process is performed.
• Restrictions on the alibi fullmask scenario
  • innerPrecise is 0.
  • pseShiftOptional is not empty, and its shape is [B, N, maxQ, maxKV].
  • attenMaskOptional is empty.
  • sparseMode is 0.

[object Object][object Object][object Object][object Object][object Object]

• The prerequisite for enabling paged attention is that [object Object] exists and is valid, and [object Object] and [object Object] are arranged in a continuous memory based on the indexes in [object Object]. In this scenario, [object Object] of [object Object] and [object Object] is invalid.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object][object Object]

NOTE

[object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object][object Object]

• When both fake-quantization parameters and KV separation quantization parameters are passed, the KV separation quantization parameters take effect.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object][object Object]

• Both [object Object] and [object Object] must be passed.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object][object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object][object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  [object Object]

[object Object][object Object]

The following example is for reference only. For details, see .