[object Object][object Object][object Object]undefined[object Object]

• Description: Performs KV compression in the Native Sparse Attention (NSA) inference phase. A new token is generated for each batch during each inference. When the number of tokens in a batch reaches the size of a compression block, the operator compresses the last compression-block-sized tokens in the batch into a compressed token.
• Formulas:

$$compressIdx=(s-compressBlockSize)/stride\\ outputCacheRef[slotMapping[i]] = input[compressIdx*stride : compressIdx*stride+compressBlockSize]*weight[:]$$ [object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.
• [object Object] and [object Object] of [object Object] are the same as those of [object Object], and [object Object] of [object Object] must be greater than [object Object].
• In the paged attention scenario, the shape of [object Object] supports [object Object]. In other scenarios, the shape of [object Object] supports [object Object].

[object Object]

The following example is for reference only. For details, see .