[object Object][object Object][object Object]undefined[object Object]

• Description: Performs routing computation for MoE based on the computation result of .

  This API has the following function changes based on . Select a proper API based on your actual requirements.

  • Added the drop mode. In this mode, the output is processed based on [object Object]. If the output exceeds the value of [object Object], the exceeded part is not processed. If the output falls short of the value of [object Object], 0s are padded.
  • Added the optional output [object Object] in dropless mode and the optional output [object Object] in drop mode.
  • Deleted the input [object Object].
  • Added the dynamic quantization mode.

• Formula:

  1. Sort the input [object Object] to obtain the sorted result [object Object] and the corresponding index [object Object].
  $$sortedExpertIdx, sortedRowIdx=keyValueSort(expertIdx)$$
  2. Use [object Object] for location mapping to obtain [object Object].
  $$expandedRowIdxOut[sortedRowIdx[i]]=i$$
  3. In dropless mode, collect statistics on the histogram of each expert in [object Object] and perform Cumsum to obtain [object Object].
  $$expertTokensCountOrCumsumOutOptional[i]=Cumsum(Histogram(sortedExpertIdx))$$
  4. In drop mode, collect statistics on the histogram of each expert in [object Object] to obtain [object Object].
  $$expertTokensBeforeCapacityOutOptional[i]=Histogram(sortedExpertIdx)$$
  5. Obtain the quantization result.

  • Static quantization:

    $$quantResult = round((x * scaleOptional) + offsetOptional)$$

  • Dynamic quantization:

    • If [object Object] is not specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$

    • If [object Object] is specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x * scaleOptional)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$
  6. Obtain the values of the first NUM_ROWS [object Object] values for [object Object] to obtain [object Object].
  $$expandedXOut[i]=quantResult[sortedRowIdx[i]\%NUM\_ROWS]$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

• [object Object]
• [object Object]

[object Object]

• Parameters:

  • [object Object] (aclTensor*, computation input): MoE input, that is, token feature input. It must be a 2D tensor with shape [NUM_ROWS, H], where [object Object] indicates the length of each token. The data type can be FLOAT16, BFLOAT16, or FLOAT32. The must be ND. are supported.
  • [object Object] (aclTensor*, computation input): K experts corresponding to features in each row of the output of . The value must be 2D shape [NUM_ROWS, K]. The data type can be INT32. The must be ND. are supported. In drop/pad scenarios or when [object Object] needs to be output in drop/pad-less scenarios, the value range must be [0, expertNum – 1]. In other scenarios, the value must be greater than or equal to [object Object].
  • [object Object] (aclTensor*, computation input): used to compute the quantization result. This parameter is optional, but mandatory in static quantization scenarios. The value is a 1D shape [1,]. In dynamic quantization scenarios, if this parameter is not set, [object Object] is not used during computation. If this parameter is set, the value must be a 2D tensor with shape [expertNum, H] or [1, H]. The data type can be FLOAT32. The must be ND. are supported.
  • [object Object] (aclTensor*, computation input): calculates the offset of the quantization result. This parameter is optional, but mandatory in static quantization scenarios. The value is a 1D shape [1,]. The data type can be FLOAT32. The must be ND. are supported.
  • [object Object] (int64_t, computation input): indicates whether the active scenario is involved. This attribute is valid only when [object Object] is set to [object Object]. The value must be greater than or equal to [object Object]. The value [object Object] indicates the dropless scenario. A value greater than [object Object] indicates the active scenario, which restricts the total number of tokens processed by all experts.
  • [object Object] (int64_t, computation input): number of tokens that can be processed by each expert. The value is greater than or equal to 0. In the drop/pad scenario, the value range is (0, NUM_ROWS]. In this case, experts drop the tokens that exceed the capacity threshold. If the capacity threshold is not reached, pad all-zero tokens. In other scenarios, this attribute value is not concerned.
  • [object Object] (int64_t, computation input): number of experts. The value is greater than or equal to [object Object]. In the drop/pad scenario or when the value of [object Object] is greater than [object Object] and the output [object Object] is required, the value of [object Object] must be greater than [object Object].
  • [object Object] (int64_t, computation input): indicates whether the scenario is in drop/pad mode. The values are [object Object] or [object Object].
    • [object Object]: drop/pad-less scenario where [object Object] is not verified.
    • [object Object]: drop/pad scenario where [object Object] and [object Object] need to be verified. The corresponding measure will be taken when the number of tokens that can be processed by each expert exceeds or falls short of the [object Object] value.
  • [object Object] (int64_t, computation input): The value can be [object Object], [object Object], or [object Object].
    • [object Object]: [object Object] is not output.
    • [object Object]: The output value is the cumulative number of tokens processed by each expert.
    • [object Object]: The output value is the number of tokens processed by each expert.
  • [object Object] (bool, computation input): The value can be [object Object] or [object Object].
    • [object Object]: [object Object] is not output.
    • [object Object]: The output value is the number of tokens processed by each expert before the drop operation.
  • [object Object] (int64_t, computation input): The values are [object Object] or [object Object].
    • [object Object]: static quantization scenario.
    • [object Object]: dynamic quantization scenario.
  • [object Object] (aclTensor*, computation output): feature expanded based on [object Object]. In the dropless/active scenario, it must be a 2D tensor. In the dropless scenario, the shape is [NUM_ROWS * K, H]. In the active scenario, the shape is [min(activeNum, NUM_ROWS * K), H]. In the drop/pad scenario, the value must be a 3D tensor with shape [expertNum, expertCapacity, H]. The data type is INT8. The must be ND. are not supported.
  • [object Object] (aclTensor*, computation output): index mapping between [object Object] and [object Object]. The value must be a 1D tensor with shape [NUM_ROWS*K, ]. The data type is INT32. The must be ND. are not supported.
  • [object Object] (aclTensor*, computation output): outputs the statistics and cumulative value of the number of tokens processed by each expert. This output is optional. The [object Object] parameter determines whether to output the value. The value is output only in drop/pad-less scenarios and must be a 1D tensor with shape [expertNum, ]. The data type can be INT32. The must be ND. are not supported.
  • [object Object] (aclTensor*, computation output): outputs the statistics on the number of tokens processed by each expert before the drop operation. This output is optional. The [object Object] parameter determines whether to output the value. The value is output only in drop/pad scenarios and must be a 1D tensor with shape [expertNum, ]. The data type can be INT32. The must be ND. are not supported.
  • [object Object] (aclTensor*, computation output): outputs the intermediate value during dynamic quantization. This output is optional. The value is output only in dynamic quantization scenarios, and must be a 1D tensor. The shape is the product of all dimensions except the last dimension of [object Object]. The data type can be FLOAT32. The must be ND. are not supported.
  • [object Object] (uint64_t*, output): size of the workspace to be allocated on the device.
  • [object Object] (aclOpExecutor**, output): operator executor, containing the operator computation process.

• Returns

  [object Object] status code. For details, see .

  [object Object]

[object Object]

• Parameters:

  • [object Object] (void*, input): address of the workspace to be allocated on the device.
  • [object Object] (uint64_t, input): size of the workspace to be allocated on the device, which is obtained by calling the first-phase API [object Object].
  • [object Object] (aclOpExecutor*, input): operator executor, containing the operator computation process.
  • [object Object] (aclrtStream, input): stream for executing the task.

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:
  • [object Object] defaults to deterministic implementation.

[object Object]

The following example is for reference only. For details, see .