[object Object][object Object][object Object]undefined[object Object]

• API function: Performs RainFusionAttention sparse attention computation with flexible block-level sparsity patterns, achieving efficient sparse attention by using [object Object] to designate which key-value blocks are selected for each query block.

• Formula:

  Sparse block size: $blockShapeX \times blockShapeY$. selectIdx specifies the sparsity mode.

  $$attentionOut = Softmax(scale \cdot query \cdot key^T + atten\_mask) \cdot value$$

  RainFusionAttention input tensors ([object Object], [object Object], and [object Object]) support flexible data layouts interpretable across multiple dimensions. Use [object Object] and [object Object] parameters to specify the desired format.

  • B: input batch size
  • T: total token length with combined B and S dimensions
  • S: sequence length of input samples
  • H: hidden layer size (Head-Size)
  • N: number of heads (Head-Num)
  • D: minimum unit size of the hidden layer (D=H/N)

  The following layouts are supported:

  • [object Object]: TND and BNSD
  • [object Object]: TND and BNSD

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:
  • [object Object] defaults to a deterministic implementation.
• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.
• Currently, [object Object] can only be TND or BNSD.
• Currently, [object Object] can only be TND or BNSD.
• The input query, key, and value must be of the same data type, which can be either FLOAT16 or BFLOAT16.
• [object Object] must contain at least two elements [blockShapeX, blockShapeY], and the element values must be greater than 0.
• The shape of [object Object] must be [T, headNum, maxKvBlockNum], where T is the total number of query blocks across all batches.
• The shape of [object Object] must be [T, headNum].
• [object Object] must be 0 (for float32 Softmax) or 1 (for fp16 Softmax). If the [object Object] input is BFLOAT16, [object Object] can only be set to 0.
• [object Object]n and [object Object] do not need to be exactly divided by [object Object]. When they are not divisible, the actual number of blocks is determined by ceiling division.
• [object Object] is required when [object Object] is TND or BNSD. [object Object] is required when [object Object] is TND or BNSD.
• Sparse block indexes must be within the valid range. Fill invalid positions with -1.
• If headNum of input [object Object] is N1 and headNum of input [object Object] and [object Object] is N2, then [object Object].
• Assume G = N1/N2. G must meet the following constraint: [object Object].

[object Object]

The following example is for reference only. For details, see .