[object Object]

[object Object][object Object]undefined[object Object]

• Description: Obtains the Top-$k$ positions for each token through a series of operations.

• Formulas:

  $$Indices=\text{Top-}k\left\{[1]_{1\times g}@\left[(W@[1]_{1\times S_{k}})\odot\text{ReLU}\left(Q_{index}@K_{index}^T\right)\right]\right\}$$

  For an index query $Q_{index} \in \mathbb{R}^{g \times d}$ corresponding to a token, given the context index key $K_{index} \in \mathbb{R}^{S_{k} \times d}$ and $W \in \mathbb{R}^{g \times 1}$. $g$ indicates the group size in Grouped-Query Attention (GQA). $d$ indicates the dimension of each head. $S_k$ indicates the context length.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

[object Object]

[object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• The value of N in the query parameter can be less than or equal to 64, and the value of N in the key parameter can be 1.
• headdim supports 128.
• The value of block_size is a multiple of 16, and the maximum value is 1024.
• The data types of [object Object] and [object Object] must be identical.
• When the data type of [object Object] is not [object Object], the data types of [object Object], [object Object], and [object Object] must be identical.

[object Object]

The following example is for reference only. For details, see .