[object Object]

[object Object][object Object]undefined[object Object]

• Description: When TP domain communication is involved, ReduceScatterV is performed first, followed by AlltoAllV, and finally the received data is aggregated (multiplied by weights and then summed). When TP domain communication is not involved, AlltoAllV is performed, after which the received data is aggregated (multiplied by weights and then summed), followed by fused Add and RMSNorm.

• Formula:

$$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ combineOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)\\ x = combineOut + residualX\\ y = \frac{x}{RMS(x)} * gamma,\quad\text{where}\ RMS(x) = \sqrt{\frac{1}{H}\sum_{i=1}^{H}x_{i}^{2}+normEps}$$

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

1. Deterministic computing:

   • [object Object] defaults to a deterministic implementation.

2. [object Object] and [object Object] must be used together. For details, see the example.

3. The values of [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] used during API calling must be the same for all ranks, at all network layers, and the same as those of [object Object].

4. [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, "this rank" in the parameter description indicates a single die.

5. The shape format is described as follows:

   • A: Maximum number of tokens to be distributed by the current rank. The value range is as follows:
     • For the shared expert, the following condition must be met: A = BsepWorldSizesharedExpertNum / sharedExpertRankNum.
     • For the MoE expert, when globalBS is 0, the following condition must be met: A >= BsepWorldSizemin(localExpertNum, K). When globalBS is not 0, the following condition must be met: A >= globalBS min(localExpertNum, K).
   • H: Hidden layer size, which must be in the range [1024, 8192].
   • Bs: Batch sequence size (number of tokens output by the rank), which must be in the range 0 < Bs ≤ 512.
   • K: Number of top K experts, which must be in the ranges 0 < K ≤ 16 and 0 < K ≤ moeExpertNum.
   • localExpertNum: Number of experts on the current rank.
     • For shared expert ranks, localExpertNum = 1.
     • For MoE expert ranks, localExpertNum = moeExpertNum/(epWorldSize - sharedExpertRankNum). If localExpertNum > 1, TP domain communication is not supported.

6. HCCL_BUFFSIZE: Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

   • In the EP communicator, the size must be greater than or equal to 2 and meet the following condition: 1024^2(HCCL_BUFFSIZE – 2)/2 ≥ Bs2(H + 128)(epWorldSize x localExpertNum + K + 1). The localExpertNum parameter must be set to the number of experts on the MoE expert card.
   • Within a TP communication domain: The value must satisfy >= (A * Align512(Align32(h * 2) + 44) + A * Align512(h * 2)) * 2.

7. Constraints on the use of communication domains:

   • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
   • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

[object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], Ascend 950PR/Ascend 950DT:

  [object Object]