[object Object]

[object Object][object Object]undefined[object Object]

When there is TP domain communication, [object Object] communication is performed first, followed by [object Object] communication, and finally the received data is combined (multiplied by the weight and then summed). When there is no TP domain communication, [object Object] communication is performed, and then the received data is combined (multiplied by the weight and then summed).

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]

    • Shared experts are not supported.
    • The shape of [object Object] is (moeExpertNum + 2 globalBs K serverNum, ), where K indicates the number of top K experts, [object Object] indicates the number of tokens received from each rank in the EP communication domain, and 2 globalBs K [object Object] indicates the number of tokens and communication area offset that can be combined for the reduce operation before communication across storage servers or within a storage server. If [object Object] is 0, the shape is calculated as follows: Bs * epWorldSize.
    • Currently, TP domain communication is not supported.
    • [object Object] must be a 1D tensor with shape [object Object].
    • epWorldSize: The value is 16, 32, or 64.
    • [object Object] must also meet the following condition: moeExpertNum / (epWorldSize - sharedExpertRankNum) <= 24.
    • [object Object] is not supported in the current version. Pass an empty string.
    • The current version does not support [object Object], [object Object], [object Object], [object Object], and [object Object]. Pass 0 for these parameters.
    • If all ranks have the same batch size, globalBs = Bs epWorldSize or 0. If batch size differs across ranks, globalBs = maxBs epWorldSize or 256 epWorldSize, where maxBs is the maximum batch size of a single rank. You are advised to pass the value as maxBs epWorldSize.

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • The shape of [object Object] is (epWorldSize max(tpWorldSize, 1) localExpertNum, ).
    • When there is TP domain communication, [object Object] is a 1D tensor with shape [object Object].
    • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
    • [object Object]: The value is 8, 16, 32, 64, 128, 144, 256, or 288.
    • The string length of [object Object] must be in the range [1, 128). It cannot have the same value with [object Object].
    • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
    • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
    • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
    • The value range of [object Object] is [0, 1]. The value 0 indicates no shared expert, and the value 1 indicates one shared expert. In the current version, only the value 1 is supported.
    • The current value range of [object Object] is [0, epWorldSize). When it is not 0, it must satisfy epWorldSize % sharedExpertRankNum = 0.
    • If all ranks have the same [object Object], globalBs = Bs epWorldSize or 0. If Bs differs across ranks, globalBs = maxBs epWorldSize, where maxBs is the maximum [object Object] of a single rank.

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Return value

  [object Object]: status code. For details, see .

[object Object]

1. Deterministic computation:

   • [object Object] defaults to a deterministic implementation.

2. [object Object] and [object Object] must be used together. For details, see the examples.

3. The element values in the [object Object], [object Object], [object Object], and [object Object] tensor output of [object Object] may vary depending on the product model, communication algorithm, or version. Pass the tensors directly to the corresponding parameters of [object Object]. Other service logics of the model should not depend on the tensors.

4. The values of [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], [object Object], and [object Object] used during API calling must be the same for all ranks, at all network layers, and the same as those of [object Object].

5. The shape format is described as follows:

   • A: Maximum number of tokens to be distributed by the current rank. The value range is as follows:
     • For shared experts, A = Bs epWorldSize sharedExpertNum / sharedExpertRankNum.
     • For MoE experts, when [object Object] is 0, A >= Bs epWorldSize min(localExpertNum, K). When [object Object] is not 0, A >= globalBs * min(localExpertNum, K).
   • H: Hidden layer size.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0, 7168] and the value must be an integer multiple of 32.
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value is 7168.
   • Bs: Batch sequence size, that is, the number of tokens output by the current rank.
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0 < Bs ≤ 256).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is (0 < Bs ≤ 512).
   • K: indicates that top K experts are selected. The value must satisfy (0 < K ≤ moeExpertNum):
     • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: The value range is (0 < K ≤ 16).
     • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The value range is (0 < K ≤ 8).
   • serverNum: Number of server nodes. The value can only be 2, 4, or 8.
   • localExpertNum: Number of experts on the current rank.
     • For shared expert ranks, [object Object] = 1.
     • For MoE expert ranks, [object Object] = moeExpertNum/(epWorldSize - sharedExpertRankNum). If [object Object] > 1, TP domain communication is not supported.

6. HCCL_BUFFSIZE:

   Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

   • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]
     • The value requirement is (≥ 2 (Bs epWorldSize min(localExpertNum, K) H * sizeof(uint16) + 2MB)).
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
     • Within an EP communication domain: The value must be greater than or equal to 2 and satisfy (1024^2 (HCCL_BUFFSIZE - 2) / 2 ≥ Bs 2 (H + 128) (epWorldSize * localExpertNum + K + 1)). [object Object] must be the number of experts assigned to the current rank when using MoE.
     • Within a TP communication domain: The value must satisfy >=A (H 2 + 128) * 2.

7. HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE:

   [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: Set the environment variables [object Object] and [object Object] to reduce the amount of cross-machine communication data, which may improve operator performance. In this case, [object Object] requires (≥ moeExpertNum Bs (H sizeof(dtypeX) + 4 ((K + 7) / 8 8) sizeof(uint32)) + 4MB + 100MB). For the input parameter [object Object], only (moeExpertNum % (epWorldSize - sharedExpertRankNum) = 0) is required, and (moeExpertNum / (epWorldSize - sharedExpertRankNum) ≤ 24) is not required.

8. In the formulas in this document, / denotes integer division.

9. Constraints on the use of the communication domain:

   • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
   • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
   • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

10. Networking constraints:

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]

  • Preparing files:

    1. Create the [object Object] file and modify it according to the following instructions.

    2. Copy the project to the two servers and configure the [object Object] file based on the device IP addresses of the servers. Ensure that the [object Object] files on the two servers are the same.

    3. Install the CANN package and compile and run it.

  • About rankTable:

    1. You can configure the NPU resource information involved in collective communication through the ranktable file. For details, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information Through the Ranktable File" in .

    2. Run the [object Object] or [object Object] to query the device IP address. Then, set the JSON file following instructions in the collective communication guide.

    [object Object]

  • Environment variable settings:

    [object Object]

  • Set the number of servers: In 2-server 16-rank scenarios, set [object Object] to 2.

    [object Object]

    You do not need to set this variable in single-server 16-rank scenarios.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • Environment variable settings:

    [object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] and [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  [object Object]