[object Object]

[object Object][object Object]undefined[object Object]

• Description: When TP domain communication is involved, ReduceScatterV is performed first, followed by AllToAllV, and finally the received data is aggregated (multiplied by weights and then summed). When TP domain communication is not involved, AllToAllV is performed, after which the received data is aggregated (multiplied by weights and then summed), followed by fused Add and RMSNorm. The operator supports special expert scenarios.

• Formula:

$$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ combineOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)\\ x = combineOut + residualX\\ y = \frac{x}{RMS(x)} * gamma,\quad\text{where}\ RMS(x) = \sqrt{\frac{1}{H}\sum_{i=1}^{H}x_{i}^{2}+normEps}$$

[object Object]

Compared with the [object Object] API, this API has the following changes:

• Special expert scenarios are supported:

  • zeroExpert ≠ 0: Enabled by setting the [object Object] parameter to a value greater than 0.

    $$Moe(oriXOptional) = 0$$

  • copyExpert ≠ 0: Enabled by setting the [object Object] parameter to a value greater than 0 and setting a valid value for the [object Object] parameter.

    $$Moe(oriXOptional) = oriXOptional$$

  • constExpert ≠ 0: Enabled by setting the [object Object] parameter to a value greater than 0 and setting valid values for the [object Object], [object Object], [object Object], and [object Object] parameters.

    $$Moe(oriXOptional) = constExpertAlpha1Optional * oriXOptional + constExpertAlpha2Optional * constExpertVOptional$$

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object][object Object]
  • [object Object] must be a string of length [0, 128) and cannot be the same as [object Object]. This parameter can be left empty only when there is no TP domain communication.
  • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
  • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.[object Object]
  [object Object]
  • [object Object] is not supported in the current version. Pass an empty string.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.[object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic computing:

  • [object Object]aclnnMoeDistributeCombineAddRmsNormV2[object Object] defaults to a deterministic implementation.

• API constraints:

  • [object Object]aclnnMoeDistributeDispatchV3[object Object] and [object Object]aclnnMoeDistributeCombineAddRmsNormV2[object Object] must be used together. For details, see .

• Parameter consistency constraint:

  • The values of expertIds, xActiveMaskOptional, elasticInfoOptional, groupEp, epWorldSize, moeExpertNum, groupTp, tpWorldSize, expertShardType, sharedExpertNum, sharedExpertRankNum, globalBs, commAlg, zeroExpertNum, copyExpertNum, constExpertNum, and HCCL_BUFFSIZE used during API calling must be the same for all ranks, at all network layers, and the same as those of DispatchV3.

• Product-specific constraints

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, the "rank" in the parameter description indicates a single die.

• Shape variable constraints:

  • A: Maximum number of tokens to be distributed by the current rank. The value range is as follows:

    • When globalBs is 0, A must be greater than or equal to BS x epWorldSize x min(localExpertNum, K).
    • When globalBs is not 0, A >= globalBs * min(localExpertNum, K).

  • H: Hidden layer size, which must be in the range [1024, 8192].

  • Bs: Batch sequence size (number of tokens output by the rank), which must be in the range 0 < Bs ≤ 512.

  • K: Number of top K experts. The value range is 0 < K ≤ 16 and 0 < K ≤ moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum.

  • localExpertNum: Number of experts on the current rank.

    • For shared expert ranks, localExpertNum = 1.
    • For MoE expert ranks, localExpertNum = moeExpertNum/(epWorldSize - sharedExpertRankNum). If localExpertNum > 1, TP domain communication is not supported.

• Environment variable constraints:

  • HCCL_BUFFSIZE: Before calling this API, check whether the value of the HCCL_BUFFSIZE environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.
    • Within an EP communication domain: The value must be greater than or equal to 2 and satisfy 1024 ^ 2 (HCCL_BUFFSIZE - 2) / 2 >= BS 2 (H + 128) (epWorldSize * localExpertNum + K + 1). localExpertNum must be the number of experts assigned to the current rank when using MoE.
    • Within a TP communication domain: The value must satisfy >= (A * Align512(Align32(h * 2) + 44) + A * Align512(h * 2)) * 2.

• Constraints on the use of communication domains:

  • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
  • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

• Other constraints:

  • moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum < MAX_INT32, where the value of MAX_INT32 is 2147483647.
  • Ascend 950PR/Ascend 950DT: Dynamic scale-in is not supported.

[object Object]

Take the [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] as an example. Call the aclnnMoeDistributeDispatchV3 and aclnnMoeDistributeCombineAddRmsNormV2 APIs. For details about the compilation and execution process, see .

• The example code is as follows:

  [object Object]