[object Object]

[object Object][object Object]undefined[object Object]

• Provides highly efficient attention computations for long-sequence inference scenarios. [object Object] (SFA) reduces computational cost by computing only the critical portions of attention. However, it introduces a large amount of discrete memory access. This increases data movement overhead and impacts overall performance.

• Formulas:

  $$\text{softmax}(\frac{Q@\tilde{K}^T}{\sqrt{d_k}})@\tilde{V}$$

  $\tilde{K}$ and $\tilde{V}$ represent key and value tensors with higher importance obtained through a selection algorithm such as [object Object]. They typically feature sparse or block-sparse characteristics. $d_k$ represents the per-head dimension of $Q$ and $\tilde{K}$.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

[object Object]

[object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• This API can be used in inference scenarios.
• This API supports graph mode.
• N1 supports 1, 2, 4, 8, 16, 32, 64, and 128.
• block_size indicates the number of tokens in a block. The value of block_size must be a multiple of 16 and the maximum value is 1024.
• The value of D in the query parameter is the same as that of D in the key and value parameters, which is 512. The value of Dr in the query_rope parameter is the same as that of Dr in the key_rope parameter, which is 64.
• The data types of the query, key, and value parameters must be the same.
• sparse_block_size must be exactly divisible by block_size.

[object Object]

The following example is for reference only. For details, see .