[object Object]

[object Object][object Object]undefined[object Object]

• Description: Performs mixture of experts (MoE) routing based on the computation results of . Non-quantization, static quantization, and dynamic quantization configurations are supported. This API has the following function changes based on the V2 API . Select a proper API based on your actual requirements.

  1. The dynamic and static quantization functions are added, and the int8 quantization mode of expendX is supported.

  2. The output expertTokensBeforeCapacityOut is deleted, and the output expertTokensCountOrCumsumOut is added.

  3. The original output mode of V2 is compatible, and the key_value output format is added. The original attributes expertTokensBeforeCapacityFlag(bool) and expertTokensCountOrCumsumFlag(int) are redefined as expertsTokensNumFlag(bool) and expertTokensNumType(int), respectively. The following table describes the mapping between the output formats.

  [object Object]

• Formula:

  1. Sort the input [object Object] to obtain the sorted result [object Object] and the corresponding index [object Object].
  $$sortedExpertIdx, sortedRowIdx=keyValueSort(expertIdx,rowIdx)$$
  2. Use [object Object] for location mapping to obtain [object Object].

  • When [object Object] is [object Object], the API outputs scatter indices.

    $$expandedRowIdxOut[i]=sortedRowIdx[i]$$

  • When [object Object] is [object Object], the API outputs gather indices.

    $$expandedRowIdxOut[sortedRowIdx[i]]=i$$
  3. Compute the histogram of [object Object] for each expert to obtain [object Object].
  $$expertTokensCountOrCumsumOutOptional[i]=Histogram(sortedExpertIdx)$$
  4. If quantMode is not equal to -1, calculate the quantization result.
     • Static quantization:
     $$quantResult=round((x∗scaleOptional)+offsetOptional)$$
  • Dynamic quantization:

    • If [object Object] is not specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$

    • If [object Object] is specified:

      $$dynamicQuantScaleOutOptional = row\_max(abs(x * scaleOptional)) / 127$$ $$quantResult = round(x / dynamicQuantScaleOutOptional)$$
  5. Tokens are rearranged using scatter indices when the active expert range covers all experts. In other configurations, tokens are rearranged using gather indices. When [object Object] is set to [object Object], the number of tokens processed by each expert is padded to [object Object]. Tokens exceeding [object Object] are dropped, and insufficient tokens are padded with zeros. The resulting output [object Object] is obtained as follows:

  • Non-quantization scenarios:

    • Rearrangement using scatter indices:
    $$expandedXOut[i]=x[scatterRowIdx[i] // K]$$
    • Rearrangement using gather indices:
    $$expandedXOut[gatherRowIdx[i]]=x[i // K]$$

  • Quantization scenarios:

    • Rearrangement using scatter indices:
    $$expandedXOut[i]=quantResult[scatterRowIdx[i] // K]$$
    • Rearrangement using gather indices:
    $$expandedXOut[gatherRowIdx[i]]=quantResult[i // K]$$
  6. The valid element count of [object Object], denoted by [object Object], is calculated as the number of elements in [object Object] that fall within the range specified by [object Object].
  $$availableIdxNum = |\{x\in expertIdx| expert\_start \le x<expert\_end \ \}|$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters, the workspace size required for computation, and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters:

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following error codes may be returned.

  [object Object]

• Differences in Support for Different Products

  • The support for quantMode varies as follows:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: -1, 0, and 1 are supported.
    • Ascend 950PR/Ascend 950DT: -1, 1, 2, and 3 are supported.
  • Ascend 950PR/Ascend 950DT: Only the following values are supported:
    • activeNum can only be set to NUM_ROWS x K.
    • expertCapacity is verified but not used. That is, the number of tokens that each expert can process is not limited.
    • dropPadMode can only be set to 0.
    • expertTokensNumType can only be set to 1 or 2.
    • expertTokensNumFlag can only be set to true.

[object Object]

• Parameters:

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• This operator supports three performance profiles on the following product models. The admission conditions must be met for each profile. Otherwise, the general profile is used.

  • Products that support performance profiles:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]
  • The admission conditions of the performance profiles are as follows:[object Object]

[object Object]

The following example is for reference only. For details, see .