[object Object]

[object Object][object Object]undefined[object Object]

• API function: A fused operator of [object Object] and [object Object]. It performs a "combine" operation on the output of the [object Object] computation based on specified indices.

  Compared with aclnnGroupedMatmulFinalizeRoutingV2, this API has the following new features:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The input parameter tuningConfigOptional is added, which is used for tuning. The first value in the array indicates the expected number of tokens to be processed by each expert. Operator tiling is performed based on this expected value to achieve higher performance.
  • Ascend 950PR/Ascend 950DT: The MX quantization scenario is added. For details, see .

• Formulas:

  • 1. Group matrix multiplication (GMM):
    $$y_i=(x_i\times weight_i) * scale_i * perTokenScale_i$$
  • 2. Routing expert and expert output allocation:

    For each token j, routing and output expert allocation are performed as follows:

    $$y[ rowIndex[i] , : ] = y[rowIndex[i], :] + y_{i(j)}[ j - start_{i(j)}]$$

    $i(j)$ is the index of the expert to which token j is allocated. $y_{i(j)}[ j - start_{i(j)}]$ is the computation result of the token under the corresponding expert.

  • 3. Output fusion of shared experts:
    $$y [rowIndex[i],:] = y[rowIndex[i],:] + sharedInputWeight \times sharedInput[j, :]$$
  • 4. Output fusion of shared experts: The final output is the result of merging all expert outputs and shared expert outputs based on rowIndex. The calculation process is as follows:
    $$y[rowIndex[i],:] = \sum_{i \in \mathcal{E}[j]} y_i [j - start_i] + sharedInputWeight \times sharedInput[j, :]$$

    $\mathcal{E}[j]$ is the set of experts assigned to token j.

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • x1 supports only INT8. The shape is (m, k), where m is in the range of [1, 16 x 1024 x 8] and k is 2048.
  • x2 supports only INT4. When the input is of type INT32, the dimension is (e, k, n/8). When the input is converted to type INT4, the dimension is (e, k, n). The value range of e is [1, 256], k is 2048, and n is 7168.
  • scaleOptional supports INT64. The shape supports three dimensions, and the dimension is (e, 1, n). The value of e and n is the same as that of e and n in w.
  • biasOptional supports FLOAT32. The value of e and n is the same as that of e and n in w.
  • offsetOptional supports FLOAT32. The shape supports three dimensions, and the dimension is (e, 1, n). The value of e and n is the same as that of e and n in w.
  • perTokenScaleOptional supports FLOAT32. The shape supports one dimension, and the dimension is (m). The value of m is the same as that of m in x.
  • groupListOptional supports the same value of e in e and w.
  • sharedInputOptional supports two dimensions, and the dimension is (bsdp, n). The value of bsdp must be less than or equal to batchSize/e, and the value of n is the same as that of n in w.
  • logitOptional supports the same value of m in m and x.
  • rowIndexOptional supports the same value of m in m and x.
  • x1, x2, and groupListOptional are mandatory. scaleOptional, perTokenScaleOptional, logitOptional, rowIndexOptional, biasOptional, and sharedInputOptional are optional.

• Ascend 950PR/Ascend 950DT:

  • x1 does not support INT8.
  • x2 does not support INT4. The dimension is (e, k, n). In the case of transposition, the dimension is (e, n, k). The value range of e is [1, 1024].
  • scaleOptional supports FLOAT8_E8M0. The shape supports four dimensions, and the dimension is (e, n, ceil(k/64), 2). The data type can only be FLOAT8_E8M0. The transpose attribute must be the same as that of x2.
  • biasOptional supports BF16.
  • sharedInputOptional supports two dimensions, and the dimension is (bsdp, n). The value of bsdp is batchSize/dataParallelSize.
  • perTokenScaleOptional supports FLOAT8_E8M0. The shape supports three dimensions, and the dimension is (m, ceil(k/64), 2).
  • x1, x2, scaleOptional, pertokenScaleOptional, groupListOptional, logitOptional, and rowIndexOptional are mandatory parameters. biasOptional and sharedInputOptional are optional parameters. Currently, the offsetOptional parameter is not supported. None of the parameters supports empty tensors.
  • The first dimension (batch) and sharedInputOffset of out must be greater than or equal to 0.
  • x1 supports empty tensors with M being 0.
  • x2 supports empty tensors with N being 0.

• Returns:

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter validation. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns:

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: aclnnGroupedMatmulFinalizeRoutingV3 is implemented in non-deterministic mode by default. Deterministic implementation can be enabled by calling aclrtCtxSetSysParamOpt.
  • Ascend 950PR/Ascend 950DT: aclnnGroupedMatmulFinalizeRoutingV3 is implemented in non-deterministic mode by default. Deterministic implementation cannot be enabled by calling aclrtCtxSetSysParamOpt.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Only the fake-quantization scenario is supported.

  • The following table describes the supported input and output data type combinations.

    [object Object]undefined

  • In this scenario, [object Object] represents the result of per-channel and per-group offline fusion.

  • In this scenario, [object Object] represents the auxiliary result of offline computation. Its value must be $8 \times w \times scaleOptional$ and is accumulated in the first dimension.

  • This scenario supports symmetric quantization and asymmetric quantization. During symmetric quantization, [object Object] must be null. During asymmetric quantization, [object Object] represents the auxiliary result of offline computation, which is the result of $antiquantOffsetOptional \times scaleOptional$.

  • In this scenario, [object Object] and [object Object] must be null.

• Ascend 950PR/Ascend 950DT: Only the full quantization scenario of the MX is supported.

  • The following table describes the supported input and output data type combinations.

    [object Object]undefined

  • In the MXFP4/MXFP8 scenario, offsetOptional, antiquantScaleOptional, and antiquantOffsetOptional must be left empty.

  • In the MXFP4 scenario, k must be an even number. In the case of x2 non-transposition, n must be an even number.

  • In the MXFP4/MXFP8 scenario, x2 transposition or non-transposition is supported. The transposition attributes of x2 and scale must be the same.

  • e must be less than or equal to 1024.

  • In the MXFP4 scenario, k cannot be 2.

[object Object]

The following is a call example, which is for reference only. For details about the compilation and execution processes, see [Compile and Run Sample](../common/Compile and Run Sample .md).

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  [object Object]

• Ascend 950PR/Ascend 950DT:

  [object Object]