[object Object][object Object][object Object]undefined[object Object]

• Description: The [object Object] operator is the backward operator of [object Object], additionally integrating the Loss function. The [object Object] operator filters out the TopK with the highest intrinsic correlation between [object Object] and [object Object], storing them in [object Object], thereby reducing the computational load of attention in long-sequence scenarios and accelerating the inference and training performance of long-sequence networks.

• Formula: The formula for computing the value used to take the Top-k can be expressed as:

  $$I_{t,:}=W_{t,:}@ReLU(q_{t,:}@(K_{:t,:})^T)$$

  Here, $W$ represents the weights corresponding to the $t$-th token, $q$ is the matrix obtained by concatenating the $G$ query heads corresponding to the $t$-th token, and $K$ is the $K$ matrix of the $t$-th row.

  The [object Object] will be trained separately, and the corresponding loss function is:

  $$L(I){=}\sum_tD_{KL}(p_{t,:}||Softmax(I_{t,:}))$$

  Among them, $p$ is the target distribution, obtained by summing the main attention scores across all heads and then applying L1 normalization along the context direction. $D_{KL}$ is the KL divergence, whose expression is:

  $$D_{KL}(a||b){=}\sum_ia_i\mathrm{log}{\left(\frac{a_i}{b_i}\right)}$$

  By taking the derivative, the gradient expression of Loss can be obtained:

  $$dI\mathop{{}}\nolimits_{{t,:}}=Softmax \left( I\mathop{{}}\nolimits_{{t,:}} \left) -p\mathop{{}}\nolimits_{{t,:}}\right. \right.$$

  Using the chain rule, the gradients of the weights, query, and key matrices can be calculated:

  $$dW\mathop{{}}\nolimits_{{t,:}}=dI\mathop{{}}\nolimits_{{t,:}}\text{@} \left( ReLU \left( S\mathop{{}}\nolimits_{{t,:}} \left) \left) \mathop{{}}\nolimits^{{T}}\right. \right. \right. \right.$$ $$d\mathop{{q}}\nolimits_{{t,:}}=dS\mathop{{}}\nolimits_{{t,:}}@K\mathop{{}}\nolimits_{{:t,:}}$$ $$dK\mathop{{}}\nolimits_{{:t,:}}= \left( dS\mathop{{}}\nolimits_{{t,:}} \left) \mathop{{}}\nolimits^{{T}}@q\mathop{{}}\nolimits_{{:t,:}}\right. \right.$$

  Here, S is the result of the softmax operation on the QK matrix.

[object Object][object Object]

Each operator has calls. First call [object Object] to obtain the input parameters and calculate the required workspace size based on the computation process. Then, call [object Object] to execute the computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to a non-deterministic implementation. Enabling deterministic computation through [object Object] is not supported.
• Common constraints
  • Handling scenarios where input parameters are empty:
    • If the query is an empty Tensor: return directly.
    • In public constraints, the scenario where the input parameter is empty should be consistent with FAG.
  [object Object]
• Specification constraints[object Object]
• Typical value[object Object]

[object Object]

The following example is for reference only. For details, see .