[object Object][object Object][object Object]undefined[object Object]

• Description: Performs compressed attention computation during the Native Sparse Attention (NSA) inference.
• Formulas:

[object Object]$$P_{cmp}= Softmax(scale * query · key^T) \\ attentionOut = P_{cmp} · value\\ P_{slc}[j] = \sum\limits_{m=0}^{l'/d -1} \sum\limits_{n = 0}^{l/d -1} P_{cmp} [l'/d * j -m - n]\\ P_{slc'} = \sum\limits_{g=1}^{G} P_{slc} ^g,\quad \text{where} G (group size) = \frac{\text{numHeads}}{\text{numKeyValueHeads}} \\ topkIndices = topk(P_{slc'})\\$$ [object Object][object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a deterministic implementation.

• [object Object] supports only TND and BSND inputs. [object Object] indicates the total length of all input sample sequences ([object Object] of all batches), [object Object] ([object Object]) indicates the size of an input sample batch, [object Object] ([object Object]) indicates the length of the input sample sequence, [object Object] ([object Object]) indicates the number of heads, and [object Object] ([object Object]) indicates the minimum unit size of the hidden layer.
• The upper limit of [object Object] before compression can be expressed as follows: [object Object] = ([object Object] – 1) x [object Object] + [object Object]. The following conditions must be met: [object Object] and [object Object]. If the value of [object Object] does not meet the formula [object Object] or the value of [object Object] is different from the batch dimension of [object Object], the single-token inference scenario is used by default.
• In the multi-token inference scenario, the [object Object] parameter must be passed, and the value of [object Object] must be the same as the batch dimension of [object Object]. In addition, the maximum [object Object] value of [object Object] is 4, the value of [object Object] for each batch must be less than or equal to that of [object Object]. If the value of [object Object] is different from the batch dimension of [object Object], or the value of [object Object] is less than 1 or greater than 4, the single-token inference scenario is used by default.

[object Object]

The following example is for reference only. For details, see .