[object Object]

[object Object][object Object]undefined[object Object]

• Description: Computes the backpropagation of [object Object].

• Formula:

  (1) When [object Object] is not set to [object Object]:

  $$permutedTokensGrad[outIndex[i]] = unpermutedTokensGrad[permuteTokenId[i]]$$ $$permutedProbsGrad = permutedTokensGrad * permutedTokensOptional$$ $$probsGradExpertOrder = \sum_{j=0}^{hidden\_size}(permutedProbsGrad_{i,j})$$
  • When [object Object] is set to false:
  $$probsGradOut = masked\_scatter(routingMapOptional^T,probsGradExpertOrder)$$ $$permutedProbs = probsOptional^T.masked\_select(routingMapOptional^T)$$ $$permutedTokensGradOut = permutedProbs.unsqueeze(-1) * permutedTokensGrad$$
  • When [object Object] is set to true:
  $$probsGradOut[permuteTokenId[i], outIndex[i]/capacity] = probsGradExpertOrder[outIndex[i]]$$ $$permutedProbs[outIndex[i]] = probsOptional.view(1)[i]$$ $$permutedTokensGradOut = permutedProbs * permutedTokensGrad$$

  (2) When [object Object] is set to [object Object]:

  $$permutedTokensGradOut[outIndex[i]] = unpermutedTokensGrad[permuteTokenId[i]]$$
  1. [object Object] indicates the size of the first dimension of [object Object].
  2. When dropAndPad is true, each expert can process a fixed number of tokens (capacity). The first dimension of the input [object Object] is the value specified by [object Object], which indicates the number of experts. The 0th dimension of the input [object Object] is [object Object] * [object Object]. The value of [object Object] can be computed based on the two dimensions.
  3. When dropAndPad is false, each token is fixedly processed by topK_num experts. The 0th dimension of the input [object Object] is specified by [object Object], which indicates the number of tokens. The 0th dimension of the input [object Object] is the value specified by [object Object] * [object Object]. The value of [object Object] can be computed based on the two dimensions.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor covering the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.
• tokens_num indicates the number of input tokens, hidden_size indicates the dimension of the word vector, and experts_num indicates the number of experts.
• The dropAndPad parameter is used to distinguish the following two modes: When dropAndPad is set to true, each expert can process a fixed number of tokens (capacity). When dropAndPad is set to false, each token is processed by a fixed number of topK_num experts.
• When the input probsOptional is not empty and dropAndPad is set to false,
  • topK_num must be less than or equal to 512 and experts_num.
  • experts_num must meet the (ubSize - (probTypeLen + 1) * numExpertAlign-(tokenTypeLen + 8) * 256) / (6 * tokenTypeLen + 12) >= 1 requirement, where ubSize is the size of the chip's UB space, probTypeLen is the number of bytes corresponding to the data type of the input probsOptional, tokenTypeLen is the number of bytes corresponding to the data type of the input unpermutedTokensGrad, and numExpertAlign is the result of rounding up experts_num to the nearest multiple of 32.
• When probsOptional is not empty and dropAndPad is true,
  • capacity must be less than or equal to tokens_num.
  • hidden_size must be less than or equal to 256 x (ubSize – 2080) / (8 + tokenTypeLen), where ubSize is the size of the UB space of the chip, and tokenTypeLen is the number of bytes corresponding to the data type of the input unpermutedTokensGrad.

[object Object]

The following example is for reference only. For details, see .