[object Object][object Object][object Object]undefined[object Object]

• Description: This operator is used for permutation computation in MoE. It passes tokens and expert labels as [object Object] and sorts the tokens and optional [object Object] based on routing maps after broadcasting them.
• Formula: [object Object] indicates the size of the zeroth dimension of [object Object], and [object Object] indicates the size of the first dimension of [object Object]. When [object Object] is set to [object Object]:$$expertIndex=arrange(tokens\_num).expand(expert\_num,-1)$$ $$sortedIndicesFirst=expertIndex.maskedselect(routingMap.T)$$ $$sortedIndicesOut=argSort(sortedIndicesFirst)$$ $$topK = numOutTokens // tokens\_num$$ $$outToken = topK * tokens\_num$$ $$permuteTokens[sortedIndicesOut[i]]=tokens[i//topK]$$ $$permuteProbsOutOptional=probsOptional.T.maskedselect(routingMap.T)$$ When [object Object] is set to [object Object]:$$capacity = numOutTokens // expert\_num$$ $$outToken = capacity * expert\_num$$ $$sortedIndicesOut = argsort(routingMap.T,dim=-1)[:, :capacity]$$ $$permutedTokensOut = tokens.index_select(0, sorted_indices)$$ If [object Object] is not set to [object Object]:$$robs\_T\_1D = probsOptional.T.view(-1)$$ $$indices\_dim0 = arange(num\_experts)$$ $$indices\_dim1 = sorted_indices.view(expert\_num, capacity)$$ $$indices\_1D = (indices_dim0 * tokens\_num + indices\_dim1).view(-1)$$ $$permuteProbsOutOptional = probs\_T\_1D.index_select(0, indices_1D)$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • [object Object] (aclTensor*, computation input): aclTensor on the device, which indicates input tokens, that is, [object Object] in the formula. The value must be a 2D tensor with shape (tokens_num, hidden_size). The data type can be BFLOAT16, FLOAT16, or FLOAT. The must be ND. are supported.
  • [object Object] (aclTensor*, computation input): aclTensor on the device, which indicates the mapping between tokens and experts, that is, [object Object] in the formula. The value must be a 2D shape with size (tokens_num, experts_num). The data type can be INT8 or BOOL. If the data type is INT8, the value can be [object Object] or [object Object]. If the data type is BOOL, the value can be [object Object] or [object Object]. The must be ND. are supported. In drop/pad-less mode, each row must contain topK [object Object] or [object Object] values.
  • [object Object] (aclTensor*, computation input): aclTensor on the device. It is an optional input, that is, probsOptional in the formula. The number of elements specified by this parameter must be the same as that of [object Object]. When [object Object] is left empty, the optional output [object Object] is also left empty. The data type is the same as that of [object Object]. The must be ND. are supported.
  • [object Object] (int64_t, computation input): [object Object] in the formula, which is used to compute the number of valid tokens specified by [object Object] and [object Object] in the formula.
  • [object Object] (bool, computation input): [object Object] in the formula, indicating whether to enable the [object Object] mode.
  • [object Object] (aclTensor*, computation output): aclTensor on the device, that is [object Object] in the formula. It is used to extend, sort, and filter tokens based on [object Object]. The value must be a 2D tensor with the shape ([object Object], [object Object]), that is, [object Object] in the formula. The data type is the same as that of [object Object]. The must be ND. are supported.
  • [object Object] (aclTensor*, computation output): mapping between [object Object], [object Object], and [object Object] in the formula. It is an aclTensor on the device. The value must be a 1D tensor with shape (outToken), that is, [object Object] in the formula. The data type can be INT32. The must be ND. are supported.
  • [object Object] (aclTensor*, computation output): [object Object] in the formula. It is an aclTensor on the device. It is the [object Object] that has been sorted and filtered based on [object Object]. The shape is (outToken), that is, [object Object] in the formula. The data type is the same as that of [object Object]. The must be ND. are supported.
  • [object Object] (uint64_t*, output): size of the workspace to be allocated on the device.
  • [object Object] (aclOpExecutor **, output): operator executor, containing the operator computation process.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Parameters:

  • [object Object] (void*, input): address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling the first-phase API [object Object]aclnnMoeTokenPermuteWithRoutingMapGetWorkspaceSize[object Object].
  • [object Object] (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input): stream for executing the task.

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• The values of [object Object] and [object Object] must be less than [object Object]. When [object Object] is set to [object Object], the number of 1s or [object Object] values in each row of [object Object] is fixed and less than [object Object].

[object Object]

The following example is for reference only. For details, see .