[object Object]

[object Object][object Object]undefined[object Object]

• This API is used to fuse the AlltoAllv, Permute, and GroupedMatMul operators of the routing expert, and implement parallel fusion with the MatMul operator of the shared expert. Communication is performed before computation.

• Formula:

  • Routed experts:
  $$ataOut = AlltoAllv(gmmX) \\ permuteOut = Permute(ataOut) \\ gmmY = permuteOut \times gmmWeight$$
  • Shared experts:
  $$mmY = mmX \times mmWeight$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the input parameters and compute the required workspace size based on the process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object] defaults to a deterministic implementation.

• Shape variables used in parameter descriptions:

  • BSK: Number of tokens sent by the local device, which is the sum of the sendCounts parameters. The value range is (0, 52428800).
  • H1: Hidden layer size of the routed experts. The value range is (0, 65536).
  • H2: Hidden layer size of the shared experts. The value range is (0, 12288].
  • e: number of experts on a single card. The value of e is less than or equal to 48. The maximum value of e x epWorldSize is 384.
  • N1: head_num for routed experts. The value range is (0, 65536).
  • N2: head_num for shared experts. The value range is (0, 65536).
  • BS: Batch sequence size.
  • K: Number of experts selected via Top-K. The value range of K is [2, 8].
  • A: Number of tokens received by the local device, which is the sum of the recvCounts parameters.
  • The sum of the A parameters across all devices in the EP communication domain equals the sum of the BSK parameters across all devices.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: If the communication volume of a single card is less than 2 MB, the performance may deteriorate.

[object Object]

The following example is for reference only. For details, see .

Note:

• This sample code calls some HCCL collective communication library APIs, including HcclGetCommName, HcclCommInitAll, and HcclCommDestroy. For details, see .

• This sample code uses eight devices as an example. Change the value of [object Object] based on the actual number of devices.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and Ascend 950PR/Ascend 950DT:

  [object Object]