[object Object][object Object][object Object]undefined[object Object]

• Description: Performs routing computation for MoE based on the computation result of . Both non-quantization and dynamic quantization modes are supported. This API has the following function changes based on the V2 API . Select a proper API based on your actual requirements.

  1. Added the dynamic quantization function to support the INT8 dynamic quantization output of [object Object].

  2. Added the [object Object] parameter to support [object Object] filtering within the valid range.

  3. Deleted the [object Object] attribute and the output [object Object] (replaced with [object Object]).

• Formula:

  1. Sort the input [object Object] to obtain the sorted result [object Object] and the corresponding index [object Object].
  $$sortedExpertIdx, sortedRowIdx=keyValueSort(expertIdx,rowIdx)$$
  2. Use [object Object] for location mapping to obtain [object Object].
  $$expandedRowIdxOut[sortedRowIdx[i]]=i$$
  3. In drop mode, collect statistics on the histogram of each expert in [object Object] to obtain [object Object].
  $$expertTokensCountOrCumsumOutOptional[i]=Histogram(sortedExpertIdx)$$
  4. Compute the quantization result.
  • Dynamic quantization:

    • If [object Object] is not specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$

    • If [object Object] is specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x * scaleOptional)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$
  5. Obtain the values of the first NUM_ROWS [object Object] values for [object Object] to obtain [object Object].
  $$expandedXOut[i]=quantResult[sortedRowIdx[i]\%NUM\_ROWS]$$
  6. The number of valid elements specified by [object Object] in [object Object] is equal to the number of elements within the range specified by [object Object] in [object Object].$$availableIdxNum = |\{x\in expertIdx| expert\_start \le x<expert\_end \ \}|$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters, the workspace size required for computation, and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object] status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• Input value range restrictions:

  • [object Object] is not used currently. The value must be equal to [object Object] * [object Object] upon verification.
  • [object Object] is currently not used. Non-empty verification is performed only.
  • Currently, [object Object] supports only [object Object], indicating the dropless scenario.
  • Currently, [object Object] supports only [object Object] and [object Object], indicating the count mode and key_value mode, respectively.
  • [object Object] supports only [object Object], indicating that [object Object] is output.
  • [object Object] supports only [object Object] and [object Object], indicating the dynamic quantization scenario and non-quantization scenario, respectively.

• Other restrictions: This operator supports two performance templates. To use either of the two templates, the following conditions must be met. If the conditions are not met, the general template is used.

  • To use the low-latency performance template, the following conditions must be met:

    • The input shapes of [object Object], [object Object], and [object Object] must be (1, 7168), (1, 8), and (256, 7168), respectively.
    • The data type of [object Object] must be BFLOAT16.
    • The attribute requirements are as follows: [object Object]=[0, 256]; [object Object]=[object Object]; [object Object]=[object Object]; [object Object]=[object Object]

  • To use the large-batch performance template, the following conditions must be met:

    • The value range of [object Object] is [384, 8192].
    • K=8
    • expertNum=256
    • expertEnd-expertStart<=32
    • quantMode=-1
    • rowIdxType=1
    • expertTokensNumType=1

[object Object]

The following example is for reference only. For details, see .