[object Object][object Object][object Object]undefined[object Object]

• Function: The SparselightningIndexerGradKlLoss operator is the backward operator of LightningIndexer and integrates the loss calculation function. The LightningIndexer operator selects the top K tokens with the highest intrinsic relationship between the query token and key token, and stores them in SparseIndices. This reduces the amount of attention calculation in long sequence scenarios and accelerates the inference and training performance of the network with long sequences.

• Formula: The formula for calculating the top-k value can be expressed as follows:

  $$I_{t,:}=W_{t,:}@ReLU(q_{t,:}@(K_{:t,:})^T)$$

  In the formula, $W$ is the weights corresponding to the $t$th token, $q$ is the matrix obtained by combining $G$ query heads corresponding to the $t$th token, and $K$ is the $K$ matrix of the $t$th row.

  LightningIndexer is trained separately, and the corresponding loss function is as follows:

  $$L(I){=}\sum_tD_{KL}(p_{t,:}||Softmax(I_{t,:}))$$

  In the formula, $p$ is the target distribution, which is obtained by summing up all the heads of the main attention score and then performing L1 regularization on the sum result in the context direction. $D_{KL}$ is the KL divergence, and its expression is as follows:

  $$D_{KL}(a||b){=}\sum_ia_i\mathrm{log}{\left(\frac{a_i}{b_i}\right)}$$

  The gradient expression of the loss can be obtained by derivation as follows:

  $$dI\mathop{{}}\nolimits_{{t,:}}=Softmax \left( I\mathop{{}}\nolimits_{{t,:}} \left) -p\mathop{{}}\nolimits_{{t,:}}\right. \right.$$

  The chain rule can be used to calculate the gradients of the weights, query, and key matrices.

  $$dW\mathop{{}}\nolimits_{{t,:}}=dI\mathop{{}}\nolimits_{{t,:}}\text{@} \left( ReLU \left( S\mathop{{}}\nolimits_{{t,:}} \left) \left) \mathop{{}}\nolimits^{{T}}\right. \right. \right. \right.$$ $$d\mathop{{q}}\nolimits_{{t,:}}=dS\mathop{{}}\nolimits_{{t,:}}@K\mathop{{}}\nolimits_{{:t,:}}$$ $$dK\mathop{{}}\nolimits_{{:t,:}}= \left( dS\mathop{{}}\nolimits_{{t,:}} \left) \mathop{{}}\nolimits^{{T}}@q\mathop{{}}\nolimits_{{:t,:}}\right. \right.$$

  S is the softmax result of the QK matrix.

Description The data format of [object Object], [object Object], and [object Object] can be interpreted from multiple dimensions. To be specific, B (Batch) indicates the size of an input sample batch, S (Seq-Length) indicates the length of the input sample sequence, H (Head-Size) indicates the size of the hidden layer, N (Head-Num) indicates the number of heads, and D (Head-Dim) indicates the minimum unit size of the hidden layer (D = H/N). T indicates the total length of all input sample sequences. -->

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • By default, aclnnSparseLightningIndexerGradKLLoss is implemented in non-deterministic mode. Deterministic computing cannot be enabled using aclrtCtxSetSysParamOpt.
• Common Constraints
  • Handling of the scenario where the input parameter is empty:
    • If [object Object] is an empty tensor, the result is returned directly.
    • The scenario where the input parameter is empty in the general restrictions is the same as that in the FAG.
  [object Object]
• Specification Restrictions[object Object]
• Typ.[object Object]

[object Object]

The following example is for reference only. For details, see .