[object Object][object Object][object Object]undefined[object Object]

• Description: Performs selected attention computation during the Native Sparse Attention (NSA) inference.

• Formula:

  Self-attention constructs an attention model by leveraging the relationships within the input samples. The principle assumes there is an input sample sequence $x$ of length $n$, where each element of $x$ is a $d$-dimensional vector. Each $d$-dimensional vector can be regarded as a token embedding. Such a sequence is transformed by three weight matrices to obtain three $n × d$ matrices.

  Selected attention manages the KV cache in a paged storage manner (i .e., paged cache). First, $key_{topk}$ is obtained from $key$ and $value_{topk}$ is obtained from $value$ by using $topkIndices$. The self-attention computation formula is as follows:

  $$Attention(query,key,value)=Softmax(\frac{query · key_{topk}^T}{\sqrt{d}})value_{topk}$$

  The product of $query$ and $key_{topk}^T$ represents the attention to the input $x$. To prevent this value from becoming excessively large, it is typically scaled by dividing by the square root of $d$, followed by row-wise softmax normalization. The result is then multiplied by $value_{topk}$ to produce an $n × d$ matrix.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• The B axis must be less than or equal to [object Object].
• Only paged attention is supported.
• In multi-token inference scenarios, the S axis of [object Object] can be at most [object Object] and for each batch, [object Object] must be less than or equal to [object Object].

[object Object]

The following example is for reference only. For details, see .