[object Object]

[object Object][object Object]undefined[object Object]

• Description: When there is TP domain communication, ReduceScatterV communication is performed first, followed by AllToAllV communication, and finally the received data is combined (multiplied by the weight and then summed). When there is no TP domain communication, AllToAllV communication is performed, and then the received data is combined (multiplied by the weight and then summed).
• Formulas:
  • When there is no communication in the TP domain:
  $$ataOut = AllToAllV(expandX)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$
  • When there is TP domain communication:
  $$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]

    • Shared experts are not supported.
    • The shape of [object Object] is (moeExpertNum + 2 globalBS K serverNum,), where K indicates the number of top K experts, moeExpertNum indicates the number of tokens received from each rank in the EP communication domain, and 2 globalBS K serverNum indicates the number of tokens and communication area offset that can be combined for the reduce operation before communication across storage servers or within a storage server. If [object Object] is 0, the shape is calculated as follows: BS * epWorldSize.
    • Currently, TP domain communication is not supported.
    • [object Object] must be a 1D tensor with shape (A, ).
    • epWorldSize: The value is 16, 32, or 64.
    • [object Object] must also meet the following condition: moeExpertNum / (epWorldSize - sharedExpertRankNum) <= 24.
    • [object Object] is not supported in the current version. Pass an empty string.
    • The current version does not support [object Object], [object Object], [object Object], [object Object], and [object Object]. Pass 0 for these parameters.
    • If all ranks have the same batch size, globalBS = BS epWorldSize or 0. If batch size differs across ranks, globalBS = maxBS epWorldSize or 256 epWorldSize, where maxBS is the maximum batch size of a single rank. You are advised to pass the value as maxBS epWorldSize.
    • The value of commQuantMode can be 0 or 2. 0 indicates that communication is not quantized, and 2 indicates that communication is quantized to int8. (The value 2 is supported only when HCCL_INTRA_PCIE_ENABLE is set to 1, HCCL_INTRA_ROCE_ENABLE is set to 0, and the driver version is 25.0.RC1.1 or later.)

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The shape of epSendCounts is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
    • When there is TP domain communication, [object Object] is a 1D tensor with shape (tpWorldSize,).
    • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
    • [object Object]: The value is 8, 16, 32, 64, 128, 144, 256, or 288.
    • [object Object] must be a string of length [0, 128) and cannot be the same as [object Object]. This parameter can be left empty only when there is no TP domain communication.
    • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
    • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value range of [object Object] is [0, 1]. The value 0 indicates no shared expert, and the value 1 indicates one shared expert. In the current version, only the value 1 is supported.
    • The current value range of [object Object] is [0, epWorldSize). When it is not 0, it must satisfy epWorldSize % sharedExpertRankNum = 0.
    • If the BSs of all ranks are the same, globalBS is equal to BS multiplied by epWorldSize or 0. If the BSs of all ranks are different, globalBS is equal to maxBS multiplied by epWorldSize (maxBS is the maximum BS value of a single device).
    • The value of commQuantMode can be 0 or 2. 0 indicates that communication quantization is not performed, and 2 indicates that communication quantization is performed in int8.

  • Ascend 950PR/Ascend 950DT:

    • The shape of epSendCounts is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
    • Currently, TP domain communication is not supported.
    • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
    • The value of epWorldSize can be 2, 4, 8, 16, 32, 64, 128, 144, 256, or 288.
    • [object Object] is not supported in the current version. Pass an empty string.
    • tpWorldSize is not supported in the current version. You can pass 1.
    • [object Object] is not supported in the current version. Pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value range of [object Object] is [0, 1]. The value 0 indicates no shared expert, and the value 1 indicates one shared expert. In the current version, only the value 1 is supported.
    • The current value range of [object Object] is [0, epWorldSize). When it is not 0, it must satisfy epWorldSize % sharedExpertRankNum = 0.
    • If the BSs of all ranks are the same, globalBS is equal to BS multiplied by epWorldSize or 0. If the BSs of all ranks are different, globalBS is equal to maxBS multiplied by epWorldSize (maxBS is the maximum BS value of a single device).
    • In the current version, commQuantMode can only be set to 0, indicating that communication quantization is not performed.

• Return Value

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return Value

  [object Object]: status code. For details, see .

[object Object]

1. Deterministic computation:

   • [object Object] defaults to a deterministic implementation.

2. Driver restrictions:

   • The driver versions of all nodes in the operator communicator must be the same.

3. [object Object] and [object Object] must be used together. For details, see the example.

4. The element values in the [object Object], [object Object], [object Object], and [object Object] tensor output of [object Object] may vary depending on the product model, communication algorithm, or version. Pass the tensors directly to the corresponding parameters of [object Object]. Other service logics of the model should not depend on the tensors.

5. The values of [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] used during API calling must be the same for all ranks, at all network layers, and the same as those of [object Object].

6. [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, "this rank" in the parameter description indicates a single die.

7. The shape format is described as follows:

   • A: Maximum number of tokens to be distributed by the current rank. The value range is as follows:
     • For shared experts, the following condition must be met: A = BS epWorldSize sharedExpertNum / sharedExpertRankNum.
     • For MoE experts, when globalBS is 0, the following condition must be met: A >= BS epWorldSize min(localExpertNum, K). When globalBS is not 0, the following condition must be met: A >= globalBS * min(localExpertNum, K).
   • H: Hidden layer size.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0, 7168] and the value must be an integer multiple of 32.
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and Ascend 950PR/Ascend 950DT: The value is 7168.
   • Bs: Batch sequence size, that is, the number of tokens output by the current rank.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0 < Bs ≤ 256).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and Ascend 950PR/Ascend 950DT: The value range is (0 < BS ≤ 512).
   • K: indicates that top K experts are selected. The value must satisfy (0 < K ≤ moeExpertNum):
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0 < K ≤ 16).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] and Ascend 950PR/Ascend 950DT: The value range is 0 < K ≤ 8.
   • serverNum: Number of server nodes. The value can only be 2, 4, or 8.
   • localExpertNum: Number of experts on the current rank.
     • For shared expert ranks, localExpertNum = 1.
     • For MoE expert ranks, localExpertNum = moeExpertNum/(epWorldSize - sharedExpertRankNum). If localExpertNum > 1, TP domain communication is not supported.

8. HCCL_BUFFSIZE:

   Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

   • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]
     • The size must be greater than or equal to 2 x (BS x epWorldSize x min(localExpertNum, K) x H x sizeof(uint16) + 2 MB).
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
     • Within an EP communication domain: The value must be greater than or equal to 2 and satisfy (1024^2 (HCCL_BUFFSIZE - 2) / 2 ≥ Bs 2 (H + 128) (epWorldSize * localExpertNum + K + 1)). localExpertNum must be the number of experts assigned to the current rank when using MoE.
     • Within the TP communication domain: The value must be grater than or equal to [object Object].
   • Ascend 950PR/Ascend 950DT: The value must be greater than or equal to 2 and satisfy (1024^2 (HCCL_BUFFSIZE - 2) / 2 ≥ Bs 2 (H + 128) (epWorldSize * localExpertNum + K + 1)). localExpertNum must be the number of experts assigned to the current rank when using MoE.

9. HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE: [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Set the environment variables [object Object] and [object Object] to reduce the amount of cross-machine communication data, which may improve operator performance. In this case, HCCL_BUFFSIZE requires (≥ moeExpertNum Bs (H sizeof(dtypeX) + 4 ((K + 7) / 8 8) sizeof(uint32)) + 4MB + 100MB). For the input parameter moeExpertNum, only (moeExpertNum % (epWorldSize - sharedExpertRankNum) = 0) is required, and (moeExpertNum / (epWorldSize - sharedExpertRankNum) ≤ 24) is not required.

10. In the formulas in this document, / denotes integer division.

11. Constraints on the use of the communication domain:

    • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
    • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

12. Networking constraints:

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]

  • Preparing files:

    1. Create the rank_table_m2.json file and modify it according to the following instructions.

    2. Copy the project to the two servers and configure the rank_table_m2.json file based on the device IP addresses of the servers. Ensure that the rank_table_m2.json files on the two servers are the same.

    3. Install the CANN package and compile and run it.

  • About rankTable:

    1. You can configure the NPU resource information involved in collective communication through the ranktable file. For details, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information Through the ranktable File" in .

    2. Run the [object Object] or [object Object] to query the device IP address. Then, set the JSON file following instructions in the collective communication guide.

    [object Object]

  • Environment variable settings:

    [object Object]

  • Set the number of servers: In 2-server 16-rank scenarios, set MACHINE_NUM to 2.

    [object Object]

    You do not need to set this variable in single-server 16-rank scenarios.

  • Build and run the operator. Build operators on all servers. The operator build commands are as follows. Both the moe_distribute_dispatch and moe_distribute_combine operators need to be built and executed in pairs.

    [object Object]

    Run the following command to execute the operator on all servers (in the two-server scenario, you need to execute the operator on both servers):

    [object Object]

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • Environment variable settings:

    [object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950PR/Ascend 950DT:

  [object Object]