[object Object]

[object Object][object Object]undefined[object Object]

• Description: Implements the fusion of GroupedMatMul, Unpermute, and AlltoAllv for routed experts, and achieves parallel fusion with the MatMul for shared experts. Computation is performed before communication.

• Formulas:

  • Routed experts:
  $$gmmY = gmmX \times gmmWeight \\ unpermuteOut = Unpermute(gmmY) \\ y = AlltoAllv(unpermuteOut)$$
  • Shared experts:
  $$mmY = mmX \times mmWeight$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Return

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computation:

  • [object Object] defaults to deterministic implementation.

• Shape variables used in parameter descriptions:

  • [object Object]: Number of tokens received by the local device, which is the sum of all elements in the [object Object] parameter. The value range is (0, 52428800).
  • [object Object]: Hidden layer size of the routed experts. The value range is (0, 65536).
  • [object Object]: Hidden layer size of the shared experts. The value range is (0, 12288].
  • [object Object]: Number of experts on a single device. [object Object] <= 32. The maximum supported value for [object Object] is 256.
  • [object Object]: [object Object] for routed experts. The value range is (0, 65536).
  • [object Object]: [object Object] for shared experts. The value range is (0, 65536).
  • [object Object]: Batch sequence size.
  • [object Object]: Number of experts selected via Top-K. The value range is [2, 8].
  • [object Object]: Number of tokens sent by the local device, which is the sum of all elements in the [object Object] parameter.
  • The sum of the [object Object] parameters across all devices in the EP communication domain equals the sum of the [object Object] parameters across all devices.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: If the communication volume of a single card is less than 2 MB, the performance may deteriorate.

[object Object]

The following example is for reference only. For details, see .

Note: This sample code calls some HCCL collective communication library APIs, including HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and Ascend 950PR/Ascend 950DT:

  [object Object]