[object Object][object Object][object Object]undefined[object Object]

• Description: Rearrange the data of size [object Object] based on [object Object] for key and value, then compute the backward output of attention in the training scenario.

• Formula: Based on the passed [object Object], select [object Object] pieces of data of size [object Object] from [object Object] and [object Object] for reordering. The formula is as follows:

  $$selectedKey\text{ }=\text{ }Gather \left( key,topkIndices \left[ i \left] \left) ,\text{ }0\text{ } < =i < \text{ }selectBlockCount\right. \right. \right. \right.$$ $$selectedValue\text{ }=\text{ }Gather \left( value,topkIndices \left[ i \left] \left) ,\text{ }0\text{ } < =i < \text{ }selectBlockCount\right. \right. \right. \right.$$

[object Object]

Phase 1: Compute $dP$ and $dV$ based on the matrix multiplication derivative rules:

[object Object]$$dP\mathop{{}}\nolimits_{{t,:}}=dO\mathop{{}}\nolimits_{{t,:}}\text{@}V\mathop{{}}\nolimits^{{T}}$$ $$dV \left[ u \left] =P\mathop{{}}\nolimits_{{T}}^{{t,:}}\text{@}dO\mathop{{}}\nolimits_{{t,:}}\right. \right.$$ [object Object]

Phase 2: Compute $dS$:

[object Object]$$d\mathop{{S}}\nolimits_{{t,:}}= \left[ P\mathop{{}}\nolimits_{{t,:}}@ \left( dP\mathop{{}}\nolimits_{{t,:}}-FlashSoftmaxGrad \left( dO,O \left) \left) \right] \right. \right. \right. \right.$$ [object Object]

Phase 3: Compute $dQ$ and $dK$:

[object Object]$$d\mathop{{Q}}\nolimits_{{t,:}}=d\mathop{{S}}\nolimits_{{t,:}}@K \left[ u \left] \mathop{{}}\nolimits_{{:t,:}}/\sqrt{{d\mathop{{}}\nolimits_{{k,:}}}}\right. \right.$$ $$dK \left[ u \left] \mathop{{}}\nolimits_{{:t,:}}=dS\mathop{{}}\nolimits_{{t,:t}}\mathop{{}}\nolimits^{{T}}\text{@}Q/\sqrt{{d\mathop{{}}\nolimits_{{t,:}}}}\right. \right.$$ [object Object]

Each operator has calls. First call [object Object] to obtain the required workspace size for computation and the executor that includes the operator's computation process. Then call [object Object] to perform the computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to a non-deterministic implementation. Enabling deterministic computation via [object Object] is not supported.

• Common constraints

  • Handling scenarios where input parameters are empty:
    • If the query is an empty Tensor: return directly.

• Mask

  [object Object]

• Specification constraints

  [object Object]

[object Object]

The following uses the [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] as an example. For details, see .