[object Object][object Object][object Object]undefined[object Object]

• Description: In training scenarios, FloydAttn differs from traditional FlashAttention by treating the sequence dimension ([object Object]) as an additional batch axis during QK/PV attention computation, thereby converting the attention computation into batch matrix multiplication ([object Object]).

• Formula:

  The forward propagation formula for attention is as follows:

  $$S=Mask(scale*(Q*K_1^T + Q*K_2^T), atten\_mask) \\ P=Softmax(S) \\ Y=(P*V_1+P*V_2)$$

  Then the backward propagation formula for attention is as follows:

  $$dV_1=P^TdY$$ $$dV_2=P^TdY$$ $$dQ=\frac{((dS)*K_1)}{\sqrt{d}}+\frac{((dS)*K_2)}{\sqrt{d}}$$ $$dK_1=\frac{((dS)^T*Q)}{\sqrt{d}}$$ $$dK_2=\frac{((dS)^T*Q)}{\sqrt{d}}$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• This API does not support determinism.

• When this API is used together with PyTorch, ensure that the CANN package versions match the PyTorch package versions.

• Shape constraints:

  • B: The value ranges from 1 to 2K.
  • [object Object]: The value ranges from 1 to 256.
  • [object Object]: The value ranges from 16 to 1M and must be a multiple of 16.
  • [object Object]: The value ranges from 128 to 1M and must be a multiple of 128.
  • [object Object]: The value ranges from 128 to 1M and must be a multiple of 128.
  • D: The value can be 32, 64, or 128.

• The 0th, 2nd, and 4th axes of query and key1 must be the same.

• The shapes of key1 and value1 must be the same.

• The shapes of key2 and value2 must be the same.

• The shapes of query and dy/attentionIn must be the same.

• The shapes of softmaxMax and softmaxSum must be the same.

• [object Object]: Only [object Object], [object Object], and [object Object] are supported.

• Due to the restriction of underlying instructions, when M x D >= 65536 or K x D >= 65536, the performance deteriorates significantly. In this case, you are advised to use small operators to implement the function.

[object Object]

The following example is for reference only. For details, see .