[object Object][object Object][object Object]undefined[object Object]

• Adapts to the [object Object] operator in the decode ([object Object]) and prefill ([object Object]) inference scenarios. Compared with [object Object], this API introduces new parameters: [object Object] and [object Object].

  Note:

  KV cache specific to the decode scenario: KV cache is a common technology for optimizing the inference performance of foundation models. During sampling, the transformer model uses the given prompt/context as the initial input for inference (parallel processing supported), and then generates additional tokens one by one to improve the generated sequence (reflecting the auto-regressive property of the model). The transformer performs the self-attention operation during sampling. Therefore, KV vectors need to be extracted for each item (regardless of the prompt/context or generated token) in the current sequence. These vectors are stored in a matrix called KV cache.

• Formula:

  Self-attention constructs an attention model by leveraging the relationships within the input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n × d$ matrices.

  The computation formula for self-attention is generally defined as follows, where $Q$, $K$, and $V$ are key attribute elements of the input sample, obtained through spatial transformation and unified into a single feature space. "Attention" in the formula and operator name is an abbreviation for "self-attention."

  $$Attention(Q,K,V)=Score(Q,K)V$$

  In this operator, the [object Object] function is used, instead of the [object Object] function. The self-attention computation formula is as follows:

  $$Attention(Q,K,V)=Softmax(\frac{QK^T}{\sqrt{d}})V$$

  The product of $Q$ and $K^T$ represents the attention to the input $x$. To prevent this value from becoming excessively large, it is typically scaled by dividing by the square root of $d$, followed by row-wise softmax normalization. The result is then multiplied by $V$ to produce an $n × d$ matrix.

  Note:

  [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a deterministic implementation.

• Common constraints

  • Processing when the input parameter is empty:
    • If [object Object] is an empty tensor, the result is returned directly.
    • If [object Object] is not an empty tensor but [object Object] and [object Object] are empty tensors (that is, [object Object] is [object Object]), [object Object] is filled with all zeros.
    • If [object Object] is an empty tensor, the framework will process it.
    • For input parameters which support the passing of [object Object] as described in the preceding parameter description, no processing is performed when they are null pointers.

• [object Object][object Object]Mask

  [object Object]

• [object Object][object Object]Paged attention

  • The prerequisite for enabling paged attention is that [object Object] exists and is valid, and [object Object] and [object Object] are arranged in a continuous memory based on the indexes in [object Object]. In this scenario, [object Object] of [object Object] and [object Object] is invalid.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

• [object Object][object Object]innerPrecise

  Note:

  [object Object][object Object]

• [object Object][object Object]pseShift

  [object Object]

• [object Object][object Object]INT8 quantization

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:[object Object]

• [object Object][object Object]Fake-quantization parameters

  • When both fake-quantization parameters and KV separation quantization parameters are passed, the KV separation quantization parameters take effect.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

• [object Object][object Object]Restrictions on [object Object], [object Object], and [object Object] in the [object Object], [object Object], and [object Object] scenarios

  • Both [object Object] and [object Object] must be passed.

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

• [object Object][object Object]MLA ([object Object]queryRope[object Object] and [object Object]keyRope[object Object] not null)

  [object Object]

• [object Object][object Object]Constraints on five-dimensional shapes of [object Object]key[object Object]/[object Object]value[object Object] in GQA/MHA/MQA fake-quantization scenario:

  [object Object]

• When [object Object] is greater than [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

• When [object Object] is equal to [object Object] (IFA non-MTP)

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:[object Object]

[object Object]

The following example is for reference only. For details, see .