[object Object]

[object Object][object Object]undefined[object Object]

• Description: Quantizes token data (optional). When there is TP domain communication, AllToAllV communication in the EP domain is performed first, and then AllGatherV TP domain communication is performed. When there is no such communication, AllToAllV communication in the EP domain is performed.

• Formulas:

  • Case 1: If quantMode = 0 (non-quantization scenario):
  $$allToAllXOut = AllToAllV(X)\\ expandXOut = \begin{cases} AllToAllV(X), & No TP communication domain AllGatherV(allToAllXOut), & TP communication domain \end{cases}$$
  • Case 2: If quantMode = 2 (pertoken dynamic quantization scenario):
  $$xFp32 = CastToFp32(X) \times scales \\ dynamicScales = dstTypeMax/Max(Abs(xFp32)) \\ quantOut = CastToInt8(xFp32 \times dynamicScalesValue) \\ allToAllXOut = AllToAllV(quantOut) \\ allToAllDynamicScalesOut = AllToAllV(1.0/dynamicScales) \\ expandXOut = \begin{cases} AllToAllV(X), & No TP communication domain\\ AllGatherV(allToAllXOut), & With TP communication domain\\ \end{cases} \\ dynamicScalesOut = \begin{cases} AllGatherV(allToAllDynamicScalesOut), & No TP communication domain\\ allToAllDynamicScalesOut, & With TP communication domain\\ \end{cases}$$

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    • The expert sharing scenario is not supported. The expertShardType, sharedExpertNum, and sharedExpertRankNum attributes are not supported.
    • Only the EP domain is supported. There is no TP domain. The groupTp, tpWorldSize, and tpRankId attributes are not supported. The tpRecvCounts attribute is invalid.
    • The expandScales content is valid only when the environment variables HCCL_INTRA_PCIE_ENABLE = 1 and HCCL_INTRA_ROCE_ENABLE = 0 are set.

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

    • expandScales is not supported.
    • The xActiveMask input is not supported.
    • The value range of sharedExpertNum is [0, 1]. The value 0 indicates that no expert is shared, and the value 1 indicates that one expert is shared. In the current version, only the value 1 is supported.
    • The current value range of [object Object] is [0, epWorldSize). When it is not 0, it must satisfy epWorldSize % sharedExpertRankNum = 0.

  • Ascend 950PR/Ascend 950DT:

    • expandScales is not supported.
    • The xActiveMask input is not supported.
    • The value range of sharedExpertNum is [0, 1]. 0 indicates that there is no shared expert, and 1 indicates that there is one shared expert. In the current version, only 1 can be passed.
    • The current value range of [object Object] is [0, epWorldSize). When it is not 0, it must satisfy epWorldSize % sharedExpertRankNum = 0.
    • Currently, communication in the TP domain is not supported. The groupTp, tpWorldSize, and tpRankId attributes are not supported. In addition, tpSendCounts is invalid.

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters:

  [object Object]

[object Object]

• Deterministic computing:

  • aclnnMoeDistributeDispatch defaults to a deterministic implementation.

• Driver restrictions:

  • The driver versions of all nodes in the operator communicator must be the same.

• [object Object] and [object Object] must be used together. For details, see the example.

• The element values in the [object Object], [object Object], [object Object], and [object Object] tensor output of [object Object] may vary depending on the product model, communication algorithm, or version. Pass the tensors directly to the corresponding parameters of [object Object]. Other service logics of the model should not depend on the tensors.

• The values of the groupEp, epWorldSize, moeExpertNum, groupTp, tpWorldSize, expertShardType, sharedExpertNum, sharedExpertRankNum and globalBS attributes used during operator calling must be consistent on all cards and at different layers on the network, and be consistent with those of the corresponding parameters of the MoeDistributeCombine operator.

• The shape format is described as follows:

  • [object Object]: Maximum number of tokens that can be received by the current rank. The value range is as follows:
    • For shared experts, A = BS * epWorldSize * sharedExpertNum / sharedExpertRankNum.
    • For the MoE expert, when globalBS is 0, the following condition must be met: A >= BS epWorldSize min(localExpertNum, K). When globalBS is not 0, the following condition must be met: A >= globalBS * min(localExpertNum, K).
  • [object Object]: Number of experts on the current rank.
    • For shared expert ranks, [object Object] = 1.
    • For MoE expert ranks, [object Object] = [object Object] / ([object Object] - [object Object]). If [object Object] > 1, TP domain communication is not supported.

• In the formulas in this document, [object Object] denotes integer division.

• Constraints on the use of communication domains:

  • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
  • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • The shape format is described as follows:
    • [object Object]: Size of the hidden layer. The value must be in the range (0, 7168] and be an integer multiple of 32.
    • [object Object]: Batch sequence size (number of tokens output by the rank), which must be in the range [1, 256].
    • [object Object]: indicates that top K experts are selected. The value must satisfy 0 < [object Object] ≤ moeExpertNum and be in the range [1, 16].
  • HCCL_BUFFSIZE: Before calling this operator, check whether the value of the HCCL_BUFFSIZE environment variable is proper. The environment variable indicates the size of the memory occupied by a single communication domain, in MB. If the environment variable is not set, the default value 200 MB is used. The value must satisfy >= 2 (BS epWorldSize min(localExpertNum, K) H * sizeof(uint16) + 2MB).
  • [object Object] and [object Object]: Set the environment variables [object Object] to 1 and [object Object] to 0 to reduce cross-server communication and improve the operator performance. In this case, HCCL_BUFFSIZE >= moeExpertNum BS (H sizeof(dtypeX) + 4 ((K + 7) / 8 8) sizeof(uint32)) + 4MB + 100MB. In addition, for the input parameter [object Object], only [object Object] % [object Object] = 0 is required, and [object Object] / [object Object] <= 24 is not required. However, the [object Object] feature is not supported.
  • [object Object]: The value is 16, 32, or 64.
  • [object Object] constraints:
    • If [object Object] is set to 2, it indicates the pertoken dynamic quantization scenario. The data type of [object Object] can be [object Object].
      • For [object Object], you can pass a null pointer.
      • If valid data is passed to [object Object], the shape is ([object Object], [object Object]).
  • Networking constraints: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • In this scenario, a single rank contains dual dies. Therefore, the "rank" in the parameter description indicates a single die.
  • The shape format is described as follows:
    • [object Object]: Hidden layer size, which is 7168.
    • [object Object]: Batch sequence size (number of tokens output by the rank), which must be in the range [1, 512].
    • [object Object]: indicates that top K experts are selected. The value must satisfy 0 < [object Object] ≤ moeExpertNum and be in the range [1, 8].
  • [object Object]: Before calling this operator, check whether the value of the HCCL_BUFFSIZE environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.
    • Within an EP communication domain: The value must be greater than or equal to 2 and satisfy 1024 ^ 2 (HCCL_BUFFSIZE - 2) / 2 >= BS 2 (H + 128) (epWorldSize * localExpertNum + K + 1). localExpertNum must be the number of experts assigned to the current rank when using MoE.
    • Within the TP communication domain: The value must be grater than or equal to [object Object].
  • [object Object]: The value is 8, 16, 32, 64, 128, 144, 256, or 288.
  • [object Object] constraints:
    • If [object Object] is set to 2, it indicates the pertoken dynamic quantization scenario. The data type of [object Object] can be [object Object].
      • For [object Object], you can pass a null pointer.
      • If valid data is passed to [object Object] and shared expert ranks exist, the shape is ([object Object] + [object Object], [object Object]).
      • If valid data is passed to [object Object] but no shared expert ranks exist, the shape is ([object Object], [object Object]).

• Ascend 950PR/Ascend 950DT:

  • The shape format is described as follows:
    • [object Object]: Hidden layer size, which is 7168.
    • [object Object]: Batch sequence size (number of tokens output by the rank), which must be in the range [1, 512].
    • K: indicates the number of top K experts to be selected. The value range is [1, 8] and 0 < K ≤ moeExpertNum must be met.
  • epWorldSize: The value can be 2, 4, 8, 16, 32, 64, 128, 144, 256, or 288.
  • HCCL_BUFFSIZE: Before calling this operator, check whether the value of the HCCL_BUFFSIZE environment variable is proper. This environment variable indicates the memory size occupied by a single communication domain, in MB. If this environment variable is not configured, the default value 200 MB is used. The value must be greater than or equal to 2 and meet the 1024 ^ 2 * (HCCL_BUFFSIZE - 2) / 2 >= BS * 2 * (H + 128) * (epWorldSize * localExpertNum + K + 1) requirement. The value of localExpertNum must be the number of experts on the MoE expert card.
  • [object Object] constraints:
    • When quantMode is set to 0, it indicates the non-quantization scenario. The data type of expandX can be FLOAT16 or BFLOAT16.
      • When the data type of expandX is FLOAT16 or BFLOAT16, the input scales must be a null pointer.
    • If [object Object] is set to 2, it indicates the pertoken dynamic quantization scenario. The data type of [object Object] can be [object Object].
      • For [object Object], you can pass a null pointer.
      • If valid data is passed to [object Object] and shared expert ranks exist, the shape is ([object Object] + [object Object], [object Object]).
      • If valid data is passed to [object Object] but no shared expert ranks exist, the shape is ([object Object], [object Object]).

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  • Preparing files:

    1. Create rank_table_m2.json and modify it.

    2. Copy the project to the two servers and configure the rank_table_m2.json file based on the device IP address. Ensure that the rank_table_m2.json files on the two servers are the same.

    3. Install the CANN package and compile and run it.

  • About rankTable:

    1. You can configure the NPU resource information involved in collective communication through the ranktable file. For details, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information Through the ranktable File" in .

    2. Run the [object Object] or [object Object] to query the device IP address. Then, set the JSON file following instructions in the collective communication guide.

    [object Object]

  • Environment variable settings:

    [object Object]

  • Set the number of servers: In 2-server 16-rank scenarios, set [object Object] to 2.

    [object Object]

    You do not need to set this variable in single-server 16-rank scenarios.

  • Build and run the operator. Build operators on all servers. The operator build commands are as follows. Both the moe_distribute_dispatch and moe_distribute_combine operators need to be built and executed in pairs.

    [object Object]

    Run the following command to execute the operator on all servers (in the two-server scenario, you need to execute the operator on both servers):

    [object Object]

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  • Environment variable settings:

    [object Object]

The following example is for reference only. For details, see .

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object], [object Object]Atlas A3 training products/Atlas A3 inference products[object Object], and Ascend 950PR/Ascend 950DT:

  [object Object]