[object Object][object Object][object Object]undefined[object Object]

• Description: Calculates the compress attention and select topk indexes in NSA. For details, see the paper ().

• Computation formulas: compression block size: $l$; selection block size: $l'$; and compression stride: $d$

$$P_{cmp} = Softmax(query*key^T) \\$$ $$attentionOut = Softmax(atten\_mask(scale*query*key^T, atten\_mask))*value$$ $$P_{slc}[j] = \sum_{m=0}^{l'/d-1}\sum_{n=0}^{l/d-1}P_{cmp} [l'/d*j-m-n],$$ $$P_{slc'} = \sum_{h=1}^{H}P_{slc}^{h}$$ $$P_{slc'} = topk\_mask(P_{slc'})$$ $$topkIndices = topk(P_{slc'})$$

NsaCompressAttention input tensors ([object Object], [object Object], and [object Object]) support flexible data layouts interpretable across multiple dimensions. Use [object Object] to specify the desired format (only [object Object] is supported).

• [object Object] ([object Object]): input batch size
• [object Object]: total token length with combined B and S dimensions
• [object Object] ([object Object]): sequence length of input samples
• [object Object] ([object Object]): hidden-layer size
• [object Object] ([object Object]): number of heads
• [object Object] ([object Object]): minimum unit size of the hidden layer ([object Object] = [object Object]/[object Object])

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation flow. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.
• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.
• The values of [object Object], [object Object], and [object Object] must be integer multiples of 16 and meet the following requirements: [object Object].
• compressBlockSize: 16-byte aligned, up to 128
• compressStride: 16-byte aligned, up to 64
• selectBlockSize: 16-byte aligned, up to 128
• selectBlockCount: supports [1-32] && selectBlockCount <= min(SelSkv).
• [object Object], [object Object], and [object Object] must use the cumulative sum mode and must be passed in [object Object] format.
• Due to the UB restriction, [object Object] must be less than or equal to 14000.
• [object Object]
• Currently, layoutOptional supports only TND.
• The input data types of [object Object], [object Object], and [object Object] must be the same.
• The batch sizes of the input query, key, and value must be the same.
• The headDim of the input query, key, and value must meet the following requirement: qD == kD && kD >= vD
• [object Object] of the input [object Object], [object Object], and [object Object] must be the same.
• If [object Object] of input [object Object] is N1 and headNum of input [object Object] and [object Object] is N2, then [object Object].
• Assume G = N1/N2. G must meet the following constraint: [object Object].
• The usage of [object Object] and [object Object] must comply with the description in the paper.

[object Object]

The following example is for reference only (using [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] as examples). For details, see .