[object Object]

[object Object][object Object]undefined[object Object]

• Description: When there is TP domain communication, ReduceScatterV communication is performed first, followed by AllToAllV communication, and finally the received data is combined (multiplied by the weight and then summed). When there is no TP domain communication, AllToAllV communication is performed, and then the received data is combined (multiplied by the weight and then summed).

  Compared with the [object Object] API, this API has the following changes:

  • Inputs more detailed token information to assist [object Object] in performing efficient all-rank synchronization. Therefore, the [object Object] input parameter (shape (Bs K,)) in the original API is replaced by the [object Object] parameter (shape (A 128,)).
  • Adds the [object Object] input parameter. When [object Object] is set to 0, users can provide tokens computed by shared experts.
  • Adds the [object Object] parameter to replace the [object Object] and [object Object] environment variables.

  For details, see the following parameter description.

• Formula:

$$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • Shared experts are not supported.
    • The shape of [object Object] is (moeExpertNum + 2 globalBs K serverNum, ), where K indicates the number of top K experts, moeExpertNum indicates the number of tokens received from each rank in the EP communication domain, and 2 globalBs K serverNum indicates the number of tokens and communication area offset that can be combined for the reduce operation before communication across storage servers or within a storage server. If [object Object] is 0, the shape is calculated as follows: Bs * epWorldSize.
    • Currently, TP domain communication is not supported.
    • [object Object] depends on the [object Object] value: For [object Object], it requires a 1D tensor with shape (Bs, ), where [object Object] must precede [object Object] (for example, {true, false, true} is invalid); for [object Object], it is currently not supported and a null pointer should be passed.
    • [object Object] must be a 1D tensor with shape (A, ).
    • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
    • The value of [object Object] depends on the [object Object] value: For [object Object], it supports 16, 32, 64, 128, and 256; for [object Object], it supports 16, 32, and 64.
    • The value of [object Object] must be in the range (0, 512] and satisfy moeExpertNum / (epWorldSize - sharedExpertRankNum) <= 24.
    • [object Object] is not supported in the current version. Pass an empty string.
    • [object Object] is not supported in the current version. Pass 0.
    • [object Object] is not supported in the current version. Pass 0.
    • [object Object] is not supported in the current version. Pass 0.
    • [object Object] is not supported in the current version. Pass 0.
    • [object Object] is not supported in the current version. Pass 0.
    • The value of [object Object] is 2. It is supported only when [object Object] is [object Object] or HCCL_INTRA_PCIE_ENABLE=1, HCCL_INTRA_ROCE_ENABLE=0, and the driver version is 25.0.RC1.1 or later.
    • The value of [object Object] can be [object Object], [object Object], [object Object], or [object Object]. It is recommended to use [object Object] with driver version 25.0.RC1.1 or later. When set to [object Object] or [object Object], the communication algorithm is selected based on HCCL environment variables (not recommended). [object Object] indicates that tokens are directly transmitted through RDMA. [object Object] indicates a two-stage communication process: intra-server communication followed by inter-server communication, which reduces cross-server data transmission.

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The shape of [object Object] is (epWorldSize max(tpWorldSize, 1) localExpertNum, ).
    • When there is TP domain communication, [object Object] is a 1D tensor with shape (tpWorldSize, ).
    • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
    • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
    • [object Object] must be a 2D tensor with shape (Bs, H) or a 3D tensor (the product of the first two dimensions equals Bs and the third dimension equals H). This parameter is optional. When it is provided, [object Object] must be set to 0.
    • The value of [object Object] must be in the range [2, 768].
    • The value of [object Object] must be in the range (0, 1024].
    • The string length of [object Object] must be in the range [1, 128). It cannot have the same value with [object Object].
    • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
    • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value of [object Object] must be in the range [0, 4].
    • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
    • The value of [object Object] is 2, and can be enabled only when [object Object] is less than 2.
    • [object Object] is not supported in the current version. Pass a null pointer.

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  aclnnStatus status code. For details, see .

[object Object]

1. Deterministic computing:

   • [object Object] defaults to a deterministic implementation.

2. [object Object] and [object Object] must be used together. For details, see .

3. The element values in the [object Object], [object Object], [object Object], and [object Object] tensor output of [object Object] may vary depending on the product model, communication algorithm, or version. Pass the tensors directly to the corresponding parameters of [object Object]. Other service logics of the model should not depend on the element values.

4. The values of [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] used during API calling must be the same for all ranks, at all network layers, and the same as those of [object Object].

5. The shape format is described as follows:

   • A: Maximum number of tokens to be distributed by the current rank. The value range is as follows:
     • For shared experts, A = Bs epWorldSize sharedExpertNum / sharedExpertRankNum.
     • For MoE experts, when [object Object] is 0, A >= Bs epWorldSize min(localExpertNum, K). When [object Object] is not 0, A >= globalBs * min(localExpertNum, K).
   • H: Hidden layer size.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value depends on [object Object]. [object Object] must be in the range (0, 7168] and be an integer multiple of 32. If [object Object] is used and the driver version is 25.0.RC1.1 or later, the value must be in the range (0, 10*1024] and be an integer multiple of 32.
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is [1024, 8192].
   • Bs: Batch sequence size, that is, the number of tokens output by the current rank.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0 < Bs ≤ 256).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is (0 < Bs ≤ 512).
   • K: Number of top K experts, which must be in the ranges 0 < K ≤ 16 and 0 < K ≤ moeExpertNum.
   • serverNum: Number of server nodes. The value can only be 2, 4, or 8.
   • localExpertNum: Number of experts on the current rank.
     • For shared expert ranks, localExpertNum = 1.
     • For MoE expert ranks, localExpertNum = moeExpertNum/(epWorldSize - sharedExpertRankNum). If localExpertNum > 1, TP domain communication is not supported.

6. HCCL_BUFFSIZE:

   Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

   • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
     • If [object Object] is set to [object Object] or [object Object], select [object Object] or [object Object] formula based on the HCCL environment variable.
     • If [object Object] is set to [object Object], the value requirement is (≥ 2 (Bs epWorldSize min(localExpertNum, K) H * sizeof(uint16) + 2MB)).
     • If [object Object] is set to [object Object], the value requirement is (≥ moeExpertNum Bs (H sizeof(dtypeX) + 4 ((K + 7) / 8 8) sizeof(uint32)) + 4MB + 100MB), where (moeExpertNum / (epWorldSize - sharedExpertRankNum) ≤ 24) is not required.
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
     • Within an EP communication domain: The value must be greater than or equal to 2 and satisfy (≥ 2 (localExpertNum maxBs epWorldSize Align512(Align32(2 H) + 44) + (K + sharedExpertNum) maxBs Align512(2 H))) ([object Object] indicates the number of experts assigned to the current rank when using MoE; [object Object]; [object Object]).
     • Within a TP communication domain: The value must satisfy >=A (H 2 + 128) * 2.

7. HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE: [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This environment variable is not recommended. You are advised to set [object Object] to [object Object].

8. In the formulas in this document, [object Object] denotes integer division.

9. Constraints on the use of communication domains:

   • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
   • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

10. Networking constraints:

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • Preparing files:

    1. Create rank_table_m2.json and modify it.

    2. Copy the project to the two servers and configure the rank_table_m2.json file based on the device IP address. Ensure that the rank_table_m2.json files on the two servers are the same.

    3. Install the CANN package and compile and run it based on .

  • About rankTable:

    1. You can configure the NPU resource information involved in collective communication through the ranktable file. For details, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information Through the Ranktable File" in .

    2. Run the [object Object] or [object Object] command to query the device IP address. Then, set the JSON file following instructions in the collective communication guide.

    [object Object]

  • Environment variable settings:

    [object Object]

  • Set the number of servers: In 2-server 16-rank scenarios, set MACHINE_NUM to 2.

    [object Object]

    You do not need to set this variable in single-server 16-rank scenarios.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  You do not need to configure the ranktable file or the environment variables RANK_TABLE_FILE and FIRST_RANK_ID.

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  [object Object]