[object Object][object Object][object Object]undefined[object Object]

• Operator function: The AlltoAllv operator of the routing expert is fused with the GroupedMatMul operator, and the parallel fusion with the MatMul operator of the shared expert is implemented. Communication is performed before computation.

• Formula:

  • Routed experts:

    [object Object]

  • Shared experts:

    [object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• The enumerated values of gmmXQuantMode, gmmWeightQuantMode, mmXQuantMode, and mmWeightQuantMode are related to the quantization modes as follows:

  • 0: no quantization
  • 1: pertensor
  • 2: perchannel
  • 3: pertoken
  • 4: pergroup
  • 5: perblock
  • 6: mx quantization
  • 7: per-token dynamic quantization

• Returns

  [object Object]: status code. For details, see . The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • The default deterministic implementation is used in [object Object].

• Shape variables used in parameter descriptions:

  • BSK: Number of tokens sent by the local device, which is the sum of the sendCounts parameters. The value range is (0, 52428800).
  • [object Object]: Hidden layer size of the routed experts. The value range is (0, 65536).
  • [object Object]: Hidden layer size of the shared experts. The value range is (0, 12288].
  • e: indicates the number of experts on a single device. The value range is (0, 32]. The maximum value of e x epWorldSize is 256.
  • [object Object]: [object Object] for routed experts. The value range is (0, 65536).
  • [object Object]: [object Object] for shared experts. The value range is (0, 65536).
  • [object Object]: Batch sequence size.
  • [object Object]: Number of experts selected via Top-K. The value range is [2, 8].
  • A: Number of tokens received by the local device, which is the sum of the recvCounts parameters.
  • The sum of the [object Object] parameters across all devices in the EP communication domain equals the sum of the [object Object] parameters across all devices.

• Quantization parameter constraints:

  • The current version supports only per-tensor quantization.

[object Object]

The following example is for reference only. For details, see .

Note: The quantization API supports only Ascend 950PR/Ascend 950DT series. The following example is implemented based on this series.