[object Object]

[object Object][object Object]undefined[object Object]

• Description: Quantizes token data (optional). When there is TP domain communication, AllToAllV communication in the EP domain is performed first, and then AllGatherV TP domain communication is performed. When there is no such communication, AllToAllV communication in the EP domain is performed.

  Compared with the [object Object] API, this API has the following changes:

  1. Outputs more detailed token information to assist CombineV2 series operators in performing efficient all-rank synchronization. Therefore, the [object Object] output (shape (BS × K,)) in the original interface is replaced by the [object Object] parameter (shape (A × 128,)).
  2. Adds the [object Object] parameter to replace the [object Object] and [object Object] environment variables.

  For details, see the following parameter description.

• Formula:

  • Case 1: If quantMode = 0 (non-quantization scenario):
  $$allToAllXOut = AllToAllV(X)\\ expandXOut = \begin{cases} AllToAllV(X), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases}$$
  • Case 2: If quantMode = 1 (static quantization scenario):
  $$xFp32 = CastToFp32(X) \times scales \\ quantOut = Cast(xFp32, dstType) \\ allToAllXOut = AllToAllV(quantOut)\\ expandXOut = \begin{cases} AllToAllV(quantOut), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases}$$
  • Case 3: If quantMode = 2 (per-token dynamic quantization scenario):
  $$xFp32 = CastToFp32(X) \times scales \\ dynamicScales = dstTypeMax/Max(Abs(xFp32)) \\ quantOut = CastToInt8(xFp32 \times dynamicScales) \\ allToAllXOut = AllToAllV(quantOut) \\ allToAllDynamicScalesOut = AllToAllV(1.0/dynamicScales) \\ expandXOut = \begin{cases} AllToAllV(quantOut), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases} \\ dynamicScalesOut = \begin{cases} allToAllDynamicScalesOut, & No TP communication domain\\ AllGatherV(allToAllDynamicScalesOut), & With TP communication domain\\ \end{cases}$$
  • Case 4: If quantMode = 3 (per-tile dynamic quantization scenario):
  $$xFp32 = CastToFp32(X) \times scales \\ dynamicScales = dstTypeMax/Max(Abs(xFp32)) \\ quantOut = CastToInt8(xFp32 \times dynamicScales) \\ allToAllXOut = AllToAllV(quantOut) \\ allToAllDynamicScalesOut = AllToAllV(1.0/dynamicScales) \\ expandXOut = \begin{cases} AllToAllV(quantOut), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases} \\ dynamicScalesOut = \begin{cases} allToAllDynamicScalesOut, & No TP communication domain\\ AllGatherV(allToAllDynamicScalesOut), & With TP communication domain\\ \end{cases}$$
  • Case 5: If quantMode = 4 (mxfp8 quantization scenario):
  $$sharedExp = Floor(log_2(max(x))) - emax \\ dynamicScales = 2^{sharedExp} \\ quantOut = CastToFp8(X / dynamicScales) \\ allToAllXOut = AllToAllV(quantOut) \\ allToAllDynamicScalesOut = AllToAllV(1.0 / dynamicScales) \\ expandXOut = \begin{cases} AllToAllV(quantOut), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases} \\ dynamicScalesOut = \begin{cases} allToAllDynamicScalesOut, & No TP communication domain\\ AllGatherV(allToAllDynamicScalesOut), & With TP communication domain\\ \end{cases}$$

  $emax$ indicates the value of the exponent part corresponding to the maximum normal number of this type.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This API must be used together with [object Object].

• [object Object]Atlas A3 training series products/Atlas A3 inference series products[object Object] /Ascend 950PR/Ascend 950DT: This API must be used together with aclnnMoeDistributeCombineV2 or aclnnMoeDistributeCombineAddRmsNorm.

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • The value of [object Object] can be [object Object], [object Object], [object Object], or [object Object]. It is recommended to use [object Object] with driver version 25.0.RC1.1 or later. When set to [object Object] or [object Object], the communication algorithm is selected based on HCCL environment variables (not recommended). [object Object] indicates that tokens are directly transmitted through RDMA. [object Object] indicates a two-stage communication process: intra-server communication followed by inter-server communication, which reduces cross-server data transmission.
    • [object Object] must be passed as a null pointer when [object Object] is [object Object], or when HCCL_INTRA_PCIE_ENABLE=1 and HCCL_INTRA_ROCE_ENABLE=0
    • [object Object] depends on the [object Object] value: For [object Object], it requires a 1D tensor with shape (BS, ), where [object Object] must precede [object Object] (for example, {true, false, true} is invalid); for [object Object], it is currently not supported and a null pointer should be passed.
    • The value of expertScalesOptional must be a 2D tensor with the shape of (BS, K).
    • The value of epWorldSize depends on the value of commAlg. For "fullmesh", the value can be 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 192, 256, or 384. For "hierarchy", the value can be 16, 32, or 64.
    • The value of moeExpertNum ranges from (0, 512].
    • [object Object] is not supported in the current version. Pass an empty string.
    • The current version does not support [object Object], [object Object], [object Object], [object Object], and [object Object]. Pass 0 for these parameters.
    • The shape of epRecvCountsOut is (moeExpertNum + 2 globalBS K * serverNum). The first moeExpertNum elements are the number of received tokens, and the remaining elements are the reduce information before communication.
    • Currently, TP domain communication is not supported.
    • expandScalesOut must be a 1D tensor with shape (A,).
    • [object Object] supports 0 (non-quantization) and 2 (dynamic quantization).

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • [object Object] is not supported in the current version. Pass a null pointer.
    • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
    • [object Object] is not supported in the current version. Pass a null pointer.
    • The value of [object Object] must be in the range [2, 768].
    • The value of [object Object] must be in the range (0, 1024].
    • [object Object] must be a string of length [0, 128) and cannot be the same as [object Object]. This parameter can be left empty only when there is no TP domain communication.
    • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
    • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value of [object Object] must be in the range [0, 4].
    • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
    • The shape of epRecvCountsOut is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
    • If there is communication in the TP domain, tpRecvCountsOut is a 1D shape tensor, and the shape is (tpWorldSize,).
    • [object Object] is not supported in the current version.
    • [object Object] supports 0 (non-quantization) and 2 (dynamic quantization).

  • Ascend 950PR/Ascend 950DT:

    • [object Object] is not supported in the current version. Pass a null pointer.
    • xActiveMaskOptional must be a 1D or 2D tensor. (When it is a 1D tensor, the shape is (BS, ). When it is a 2D tensor, the shape is (BS, K).) In 1D mode, true must be placed before false. For example, {true, false, true} is invalid. In 2D mode, if all the K values corresponding to a token are false, the token is not involved in communication.
    • [object Object] is not supported in the current version. Pass a null pointer.
    • The value of [object Object] must be in the range [2, 768].
    • The value of [object Object] must be in the range (0, 1024].
    • [object Object] is not supported in the current version. Pass an empty string.
    • [object Object] is not supported in the current version. Pass 0.
    • [object Object] is not supported in the current version. Pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value of [object Object] must be in the range [0, 4].
    • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
    • The shape of epRecvCountsOut is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
    • The output tpRecvCountsOut is not supported in the current version.
    • [object Object] is not supported in the current version.
    • The value of quantMode can be 0 (non-quantization), 1 (static quantization), 2 (per-token dynamic quantization), 3 (per-group dynamic quantization), or 4 (mxfp8 dynamic quantization).

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

[object Object]: status code. For details, see .

[object Object]

1. Deterministic computing:

   • [object Object] defaults to a deterministic implementation.

2. Driver restrictions:

   • The driver versions of all nodes in the operator communicator must be the same.

3. [object Object] and [object Object] must be used together. For details, see .

4. The element values in the [object Object], [object Object], [object Object], and [object Object] tensor output of [object Object] may vary depending on the product model, communication algorithm, or version. Pass the tensors directly to the corresponding parameters of [object Object]. Other service logics of the model should not depend on the element values.

5. The values of [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] used during API calling must be the same for all ranks, at all network layers, and the same as those of [object Object].

6. [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, "this rank" in the parameter description indicates a single die.

7. The shape format is described as follows:

   • A: Maximum number of tokens that can be received by the current rank. The value range is as follows:
     • For shared experts, the following condition must be met: (A = BS epWorldSize sharedExpertNum / sharedExpertRankNum)
     • For MoE experts, when globalBS is 0, the following condition must be met: (A >= BS epWorldSize min(localExpertNum, K)). When globalBS is not 0, the following condition must be met: (A >= globalBS * min(localExpertNum, K))
   • H: Hidden layer size.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value depends on commAlg. "fullmesh" must be in the range (0, 7168] and be an integer multiple of 32. If "hierarchy" is used and the driver version is 25.0.RC1.1 or later, the value must be in the range (0, 10*1024] and be an integer multiple of 32.
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is [1024, 8192].
     • Ascend 950PR/Ascend 950DT: The value is within the range of [1024, 8192].
   • Bs: Batch sequence size, that is, the number of tokens output by the current rank.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value is determined by the value of commAlg. If commAlg is set to "fullmesh", the value is within the range of (0 < BS ≤ 256). If commAlg is set to "hierarchy" and the driver version is 25.0.RC1.1 or later, the value is within the range of (0 < BS ≤ 512).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is (0 < Bs ≤ 512).
     • Ascend 950PR/Ascend 950DT: The value is within the range of (0 < BS ≤ 512).
   • K: Number of top K experts, which must be in the ranges 0 < K ≤ 16 and 0 < K ≤ moeExpertNum.
   • serverNum: Number of server nodes. The value can only be 2, 4, or 8.
   • localExpertNum: Number of experts on the current rank.
     • For shared expert ranks, localExpertNum = 1.
     • For MoE expert ranks, localExpertNum = moeExpertNum/(epWorldSize - sharedExpertRankNum). If localExpertNum > 1, TP domain communication is not supported.

8. quantMode constraints:

   • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
     • When quantMode is set to 0, it indicates the non-quantization scenario, and the input scales pointer is null.
     • If [object Object] is set to 2, it indicates the pertoken dynamic quantization scenario. The data type of [object Object] can be INT8.
       • For [object Object], you can pass a null pointer.
       • If valid data is passed to [object Object], the shape is (moeExpertNum, H).
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
     • When quantMode is set to 0, it indicates the non-quantization scenario, and the input scales pointer is null.
     • If [object Object] is set to 2, it indicates the pertoken dynamic quantization scenario. The data type of [object Object] can be INT8.
       • For [object Object], you can pass a null pointer.
       • If valid data is passed to [object Object] and shared expert ranks exist, the shape is (sharedExpertNum + moeExpertNum, H).
       • If valid data is passed to [object Object] but no shared expert ranks exist, the shape is (moeExpertNum, H).
   • Ascend 950PR/Ascend 950DT:
     • When quantMode is set to 0, it indicates the non-quantization scenario. The data type of expandX can be FLOAT16 or BFLOAT16. The input scales must be a null pointer.
     • When quantMode is set to 1, it indicates the static quantization scenario. The data type of expandX can be INT8 or HIFLOAT8.
       • When the data type of expandX is INT8, the following scenarios are supported:
         • The input scales represent the quantization coefficient, and the shape is (1,).
         • When the input scales represent the smooth weight shared by each expert, the shape is (H, ).
         • When the input scales represent the quantization coefficient that integrates the smooth weight of each expert, if there is a shared expert card, the shape is (sharedExpertNum + moeExpertNum, H); if there is no shared expert card, the shape is (moeExpertNum, H).
       • When the data type of expandX is HIFLOAT8, the shape of scales must be (1, ).
     • When quantMode is set to 2, it indicates the per-token dynamic quantization scenario. The data type of expandX can be INT8, FLOAT8_E4M3FN or FLOAT8_E5M2.
       • For [object Object], you can pass a null pointer.
       • If valid data is passed to [object Object] and shared expert ranks exist, the shape is (sharedExpertNum + moeExpertNum, H).
       • If valid data is passed to [object Object] but no shared expert ranks exist, the shape is (moeExpertNum, H).
     • When quantMode is set to 3, it indicates the per-group dynamic quantization scenario. The data type of expandX can be FLOAT8_E4M3FN or FLOAT8_E5M2.
       • For [object Object], you can pass a null pointer.
       • If valid data is passed to [object Object] and shared expert ranks exist, the shape is (sharedExpertNum + moeExpertNum, H).
       • If valid data is passed to [object Object] but no shared expert ranks exist, the shape is (moeExpertNum, H).
     • When quantMode is set to 4, it indicates the mxfp8 quantization scenario. The data type of expandX can be FLOAT8_E4M3FN or FLOAT8_E5M2. The input scales must be a null pointer.

9. HCCL_BUFFSIZE:

   Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

   • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
     • If [object Object] is set to "" or a null pointer, select the "fullmesh" or "hierarchy" formula based on the HCCL environment variable.
     • If commAlg is "fullmesh", the size must be greater than or equal to 2 x (BS x epWorldSize x min(localExpertNum, K) x H x sizeof(uint16) + 2MB).
     • If commAlg is "hierarchy", the size must be (≥ (moeExpertNum + epWorldSize / 4) * Align512(maxBS * (H * 2 + 16 * Align8(K))) * 1B + 8MB, where Align8(x) = ((x + 8 - 1) / 8) x 8, and Align512(x) = ((x + 512 - 1) / 512) x 512).
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
     • In the EP communicator, the size must be greater than or equal to 2 and meet the following condition: ≥ 2 * (localExpertNum * maxBS * epWorldSize * Align512(Align32(2 * H) + 64) + (K + sharedExpertNum) * maxBS * Align512(2 * H)). (localExpertNum indicates the number of experts on the MoE expert card. Align512(x) = ((x + 512 – 1) / 512) x 512. Align32(x) = ((x + 32 – 1) / 32) x 32.)
     • Within the TP communication domain: The value must be grater than or equal to [object Object].
   • Ascend 950PR/Ascend 950DT: The size must be greater than or equal to 2 and meet the following condition: >= aivNum * 512 + 2 * epWorldSize * (maxBS * Align512(alignedH * 2) * localExpertNum + 512). (aivNum indicates the number of cores. localExpertNum indicates the number of experts on the MoE expert card. Align512(x) = ((x + 512 – 1) / 512) x 512. The requirements for alignedH vary in different quantization scenarios.
     • In the pergroup dynamic quantization scenario, alignedH = Align128(H) = ((H + 128 – 1) / 128) x 128.
     • In the mx quantization scenario, alignedH = Align256(H) = ((H + 256 – 1) / 256) x 256.
     • In other quantization modes, alignedH = Align32(H) = ((H + 32 – 1) / 32) x 32.

10. HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE:

[object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This environment variable is not recommended. You are advised to set [object Object] to [object Object].

11. In the formulas in this document, / denotes integer division.

12. Constraints on the use of the communication domain:

    • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
    • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

13. Networking constraints:

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

[object Object]

• Preparing files:

  1. Create the [object Object] file and modify it according to the following instructions.

  2. Copy the project to the two servers and configure the [object Object] file based on the device IP addresses of the servers. Ensure that the [object Object] files on the two servers are the same.

  3. Install the CANN package and compile and run it based on .

• About rankTable:

  1. You can configure the NPU resource information involved in collective communication through the ranktable file. For details, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information Through the ranktable File" in .

  2. Run the [object Object] or [object Object] to query the device IP address. Then, set the JSON file following instructions in the collective communication guide.

  [object Object]

• Environment variable settings:

  [object Object]

• Set the number of servers:

  In 2-server 16-rank scenarios, set [object Object] to 2.

  [object Object]

  You do not need to set this variable in single-server 16-rank scenarios.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  You do not need to configure the [object Object] file or the environment variables [object Object] and [object Object].

  In this example, the A2 operator can run in a single-node environment with 2 to 8 ranks. Before running the example, set IS_TEST_A2 in the sample code to true to ensure that the A2 branch is executed. In addition, you can set EP_WORLD_SIZE_A2 to the number of ranks in the sample code and change moeExpertNum in the launchOneThreadDispatchV2AndCombineV2_A2 function so that moeExpertNum can be exactly divided by EP_WORLD_SIZE_A2.

  The operator compilation command is as follows. Both the moe_distribute_dispatch_v2 and moe_distribute_combine_v2 operators need to be compiled. These two operators must be executed in pairs.

  [object Object]

  The command for executing the sample operator is as follows:

  [object Object]

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  You do not need to configure the [object Object] file or the environment variables [object Object] and [object Object].

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950PR/Ascend 950DT:

  [object Object]