[object Object][object Object][object Object]undefined[object Object]

• Description: For [object Object] processed by [object Object], accumulates them back to the original [object Object]. This operator retrieves the input data stored in [object Object] based on the subscripts stored in [object Object]. If [object Object] data exists, [object Object] is multiplied by [object Object]. Then, this operator computes the cumulative sum and outputs the computation result.
• Formula:$$topK\_num= permutedTokens.size(0) // routingMapOptional.size(0)$$ $$numExperts = probs.size(1)$$ $$numTokens = probs.size(0)$$ $$capacity = sortedIndices.size(0) // numExperts$$ (1) When [object Object] is not set to [object Object] and [object Object] is set to [object Object]:$$permuteProbs [i//capacity,sortedIndices[i]]=probs[i]$$ $$permutedTokens = permutedTokens * permuteProbs$$ $$unpermutedTokens= zeros(restoreShape, dtype=permutedTokens.dtype, device=permutedTokens.device)$$ $$permuteTokenId, outIndex= sortedIndices.sort(dim=-1)$$ $$unpermutedTokens[permuteTokenId[i]] += permutedTokens[outIndex[i]]$$ (2) When [object Object] is not set to [object Object] and [object Object] is set to [object Object]:$$permuteProbs = probs.T.maskedSelect(routingMap.T)$$ $$permutedTokens = permutedTokens * permuteProbs$$ $$unpermutedTokens= zeros(restoreShape, dtype=permutedTokens.dtype, device=permutedTokens.device)$$ $$unpermutedTokens[i//topK\_num] += permutedTokens[sortedIndices[i]]$$ (3) When [object Object] is set to [object Object] and [object Object] is set to [object Object]:$$permuteTokenId, outIndex= sortedIndices.sort(dim=-1)$$ $$unpermutedTokens[permuteTokenId[i]] += permutedTokens[outIndex[i]]$$ (4) When [object Object] is set to [object Object] and [object Object] is set to [object Object]$$unpermutedTokens[i//topK\_num] += permutedTokens[sortedIndices[i]]$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor covering the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • [object Object] (aclTensor*, computation input): aclTensor on the device, which indicates input tokens. The value must be a 2D tensor. When [object Object] is set to [object Object], the shape is (tokens_num* topK_num, hidden_size). When [object Object] is set to [object Object], the shape is (experts_num * capacity, hidden_size). [object Object] indicates the number of tokens that can be processed by each expert. The data type can be BFLOAT16, FLOAT16, or FLOAT. The must be ND. are supported.
  • [object Object] (aclTensor*, computation input): aclTensor on the device. In drop/pad-less mode, the value must be a 1D shape with size (tokens_num * topK_num,). The data type can be INT32. The must be ND. The index value range is [0, tokens_num * topK_num – 1]. In drop/pad mode, the value must be a 1D tensor with shape (experts_num * capacity). The data type can be INT32. The must be ND. The index value range is [0, tokens_num – 1]. are supported.
  • routingMapOptional (aclTensor*, computation input): aclTensor on the device. This input is optional. If the input [object Object] is a null pointer, this input is not required, and a null pointer should be passed. In the formula, [object Object] indicates whether the token at the corresponding position is processed by the corresponding expert. The value must be a 2D shape with size (tokens_num, experts_num). The data type can be INT8 or BOOL. If the data type is INT8, the value can be [object Object] or [object Object]. If the data type is BOOL, the value can be [object Object] or [object Object]. The must be ND. are supported.
  • [object Object] (aclTensor*, computation input): aclTensor on the device. This input is optional. If it is not required, pass a null pointer. In the formula, [object Object] indicates the weight of the token at a specified position processed by the corresponding expert in the final result. The shape is the same as that of [object Object], the data type is the same as that of [object Object], and the must be ND. are supported.
  • [object Object] (bool, computation input): input of the Boolean type on the host. This input is optional. The value can be [object Object] or [object Object]. The value [object Object] indicates that [object Object] is enabled, and the value [object Object] indicates that [object Object] is disabled. When [object Object] is enabled, the shapes of the outputs [object Object] and [object Object] are (experts_num capacity,). When [object Object] is disabled, each token is processed by a fixed number of experts (specified by [object Object]), and the shapes of the outputs [object Object] and [object Object] are (tokens_num topK_num,).
  • [object Object] (aclIntArray*, computation input): aclIntArray on the host. The supported data type is INT64, and the size is [object Object]. The shape is the same as that [object Object].
  • [object Object] (aclTensor*, computation output): aclTensor on the device, that is, [object Object] in the formula. It is the forward output result. The value must be a 2D tensor with shape (tokens_num, hidden_size). The data type can be BFLOAT16, FLOAT16, or FLOAT. The must be ND. are supported.
  • [object Object] (aclTensor*, computation output): aclTensor on the device, that is, [object Object] in the formula. When [object Object] is set to [object Object], the value must be a 1D shape with size (tokens_num* topK_num,). The index value range is [0, tokens_num topK_num – 1]. When [object Object] is set to [object Object], the value must be a 1D tensor with shape (experts_num capacity,). The index value range is [0, experts_num * capacity – 1]. The data type is INT32, and the must be ND. are supported.
  • [object Object] (aclTensor*, computation output): aclTensor on the device, that is, [object Object] in the formula. When [object Object] is set to [object Object], the value must be a 1D shape with size (tokens_num* topK_num,). When [object Object] is set to [object Object], the value must be a 1D tensor with shape (experts_num * capacity,). The index value range is [0, tokens_num – 1]. The data type is INT32, and the must be ND. are supported.
  • [object Object] (aclTensor *, computation output): aclTensor on the device, that is, [object Object] in the formula, which indicates that permuted [object Object] is output. The shape can be 1D. The data type is the same as that of [object Object]. The must be ND.
  • [object Object] (uint64_t*, output): size of the workspace to be allocated on the device.
  • [object Object] (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns:

  [object Object]: status code. For details, see .

  [object Object]

[object Object]

• Parameters:

  • [object Object] (void*, input): address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling the first-phase API [object Object].
  • [object Object] (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input): stream for executing the task.

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.
• The value of [object Object] is less than or equal to [object Object]. When [object Object] is set to [object Object], the number of 1s or [object Object] values in each row of [object Object] is fixed and less than [object Object].

[object Object]

The following example is for reference only. For details, see .