[object Object][object Object][object Object]undefined[object Object]

• Description: This operator is used for permutation computation of MoE. It broadcasts and sorts tokens and optional probs based on indexes ([object Object]), and performing slicing based on the range specified by [object Object].
• Formula:

  • When [object Object] is set to [object Object],

    $$sortedIndicesFirst=argSort(indices)$$ $$sortedIndicesOut=argSort(sortedIndices)$$

    When the condition rangeOptional[0] ≤ sortedIndices[i] < rangeOptional[1] is met,

    $$permuteTokensOut[sortedIndices[i]-rangeOptional[0]]=tokens[i//topK]$$ $$permuteProbsOut[sortedIndices[i]-rangeOptional[0]]=probsOptional[i]$$

  • When [object Object] is set to [object Object],

    $$permuteTokensOut[i]=tokens[indices[i]]$$ $$sortedIndicesOut=indices$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • [object Object] (aclTensor *, computation input): input tokens in permute, that is, [object Object] in the formula, which is an aclTensor on the device. The 2D shape is supported, and the shape is (num_tokens, hidden_size). [object Object] indicates the number of tokens, and [object Object] indicates the length of each token. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The can be ND. are supported, but empty tensors are not supported.
  • [object Object] (aclTensor *, computation input): expert indexes corresponding to input tokens, that is, [object Object] in the formula, which is an aclTensor on the device. The shape supports 1D or 2D. When [object Object] is set to [object Object], it indicates the indexes of topK processing experts corresponding to each input token. The shape is (num_tokens, topK_num) or (num_tokens). When [object Object] is set to [object Object], it indicates the token indexes (not supported currently) selected by each expert. The number of elements is less than [object Object], and the value must be within the range of [object Object] to [object Object] (excluded), and the value of [object Object] must be less than or equal to [object Object]. The supported data types are INT32 and INT64. The can be ND. are supported, but empty tensors are not supported.
  • [object Object] (aclTensor *, computation input): the expert processing probability of input tokens, that is, [object Object] in the formula, which is an aclTensor on the device. Optional computation input, which corresponds to the computation output [object Object]. If this parameter is left empty, [object Object] is not output. Its shape is the same as that of [object Object]. The data type can be BFLOAT16, FLOAT16, or FLOAT32. The can be ND. are supported.
  • [object Object] (aclIntArray *, computation input): valid range of Expert Parallelism (EP) slicing. The size is [object Object] (an array of two integers). If this parameter is left empty, [object Object] and [object Object] are ignored, and the execution logic is rolled back to .
  • [object Object] (int64_t, computation input): number of valid output tokens. This parameter is valid only when [object Object] is left empty. When it is set to [object Object], no token is deleted. If this parameter is not set to [object Object], the tokens that are sorted by [object Object] and the part exceeding the value specified by [object Object] are dropped. If this parameter is set to a negative number, tokens are processed according to the rules for negative slice indexes.
  • [object Object] (bool, computation input): If [object Object] is set to [object Object], [object Object] have been padded with the token indexes selected by each expert. In this case, [object Object] is not sorted. Currently, [object Object] can only be set to [object Object].
  • [object Object] (aclTensor *, computation output): tokens that are extended and sorted based on [object Object], that is, [object Object] in the formula, which is an aclTensor on the device. The shape supports 2D, and the shape is (rangeOptional[1] – rangeOptional[0], hidden_size). The data type is the same as that of tokens. The is ND.
  • [object Object] (aclTensor *, computation output): mapping between [object Object] and [object Object], that is, [object Object] in the formula, which is an aclTensor on the device. The 1D shape is supported, and the shape is (num_tokens * topK_num). The data type is INT32, and the must be ND.
  • [object Object] (aclTensor *, computation output): probs that are extended and sorted based on indices, that is, [object Object] in the formula, which is an aclTensor on the device. The 1D shape is supported, and the shape is (rangeOptional[1] – rangeOptional[0]). The data type is the same as that of [object Object]. The must be ND.
  • [object Object] (uint64_t *, output): size of the workspace to be allocated on the device.
  • [object Object] (aclOpExecutor **, output): operator executor, containing the operator computation process.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Parameters:

  • [object Object] (void*, input): address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling the first-phase API [object Object].
  • [object Object] (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input): stream for executing the task.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• [object Object] requires that the number of elements be less than [object Object], and the value ranges from [object Object] to [object Object] (excluded). (The maximum or minimum value of int-32 or int-64 integer is supported. If the value is not within the range, the sorting result is incorrect.)

• The value of [object Object] is less than or equal to [object Object].

• [object Object] cannot be set to [object Object].

• When [object Object] is left empty, [object Object] and [object Object] are ignored, and the execution logic is rolled back to .

[object Object]

The following example is for reference only. For details, see .