[object Object]

[object Object][object Object]undefined[object Object]

• Description: When there is TP domain communication, ReduceScatterV communication is performed first, followed by AllToAllV communication, and finally the received data is combined (multiplied by the weight and then summed). When there is no TP domain communication, AllToAllV communication is performed, and then the received data is combined (multiplied by the weight and then summed).

  Compared with the [object Object] API, this API has the following changes:

  Added the capability to collect the communication duration, that is, record the communication time of each rank. This function is enabled by passing the [object Object] parameter. It is recommended that this function be used together with . The communication duration per rank for each operator call is accumulated in this tensor. Clear it as needed before use.

• Formula:

  • When there is no TP domain communication:
  $$ataOut = AllToAllV(expandX)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$
  • When there is TP domain communication:
  $$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object][object Object]
  • The value of [object Object] can be [object Object], [object Object], [object Object], or [object Object]. It is recommended to use [object Object] with driver version 25.0.RC1.1 or later. When set to [object Object] or [object Object], the communication algorithm is selected based on HCCL environment variables (not recommended). [object Object] indicates that tokens are directly transmitted through RDMA. [object Object] indicates a two-stage communication process: intra-server communication followed by inter-server communication, which reduces cross-server traffic.
  • Shared experts are not supported.
  • The shape of epSendCounts is (moeExpertNum + 2 globalBS K serverNum,). K indicates the number of top K experts. The first moeExpertNum elements indicate the number of tokens received from each card in the EP communicator. The last 2 globalBS K serverNum elements are used to store the number of tokens that can be reduced by the combine operation in advance and the offset of the communication area before inter-server/intra-server communication. When globalBS is 0, the value is calculated as follows: BS * epWorldSize.
  • Currently, TP domain communication is not supported.
  • The value of xActiveMaskOptional depends on the value of commAlg. If commAlg is set to "fullmesh", xActiveMaskOptional must be a 1D tensor with the shape of (BS,). The true value must be placed before the false value. For example, {true, false, true} is invalid. If commAlg is set to "hierarchy", xActiveMaskOptional is not supported in the current version, and a null pointer can be passed.
  • The value of expandScalesOptional must be a 1D tensor with the shape of (A, ).
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • The value of epWorldSize depends on the value of commAlg. The value of fullmesh can be 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 192, 256 or 384. The value of hierarchy can be 16, 32, or 64.
  • The value of moeExpertNum is in the range of (0, 512].
  • [object Object] is not supported in the current version. Pass an empty string.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • The value of [object Object] is 0 or 2 (0 indicates no quantization and 2 indicates int8 quantization). The value 2 is supported only when commAlg is [object Object], or when HCCL_INTRA_PCIE_ENABLE=1, HCCL_INTRA_ROCE_ENABLE=0, and the driver version is 25.0.RC1.1 or later.
  • [object Object] must be a 1D tensor.
  • [object Object] is not supported in the current version. Pass a null pointer.
  • [object Object] is not supported in the current version when commAlg is [object Object]. Pass a null pointer.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • When commAlg is [object Object], the value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [[object Object]moeExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum[object Object]).
  • When commAlg is [object Object], the value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid copy expert IDs must be in the range [[object Object]moeExpertNum + zeroExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum + copyExpertNum[object Object]).
  • [object Object] is not supported in the current version. Pass 0.
  • You can pass valid data or a null pointer for [object Object]. If you pass a null pointer, the function of recording the communication duration is disabled. If you pass valid data, it must be a 1D tensor with shape (ep_world_size,), and its data type and data format must be int64 and ND, respectively.[object Object]
  [object Object]
  • commAlg supports two input modes: "" and "hierarchy". "": The default value. The hierarchy cross-super template is disabled. "hierarchy": The hierarchy cross-super template is enabled. This template supports only the scenario where the value of tpWorldSize is 1 and the number of shared experts is 0, and does not support the variable BS, 2D mask, special expert, and performanceInfo scenarios.
  • The shape of epSendCounts is (epWorldSize max(tpWorldSize, 1) localExpertNum, ).
  • When there is TP domain communication, [object Object] is a 1D tensor with shape (tpWorldSize, ).
  • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • sharedExpertXOptional must be a 2D or 3D tensor. (When it is a 2D tensor, the shape is (BS, H). When it is a 3D tensor, the product of the first two dimensions is equal to BS, and the third dimension is equal to H.) This parameter can be passed or not. If it is passed, the value of sharedExpertRankNum must be 0.
  • The value of epWorldSize ranges from 2 to 768. When commAlg="hierarchy", the value must be an integer multiple of 16 and ranges from 16 to 256.
  • The value of moeExpertNum ranges from (0, 1024]. When commAlg="hierarchy", the value ranges from (0, 512].
  • groupTp must be a string of length [0, 128) and cannot be the same as groupEp. This parameter can be left empty only when there is no TP domain communication.
  • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
  • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
  • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
  • The value of [object Object] must be in the range [0, 4].
  • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
  • The value of [object Object] is 0 or 2 (0 indicates no quantization and 2 indicates int8 quantization). The value 2 is supported only when tpWorldSize < 2.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • elasticInfoOptional is a reserved parameter and is not supported in the current version. You can pass a null pointer.
  • constExpertAlpha1Optional can be set to a valid data or null pointer. When constExpertNum is not 0, valid input must be passed. When valid data is passed, it must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • constExpertAlpha2Optional can be set to a valid data or null pointer. When constExpertNum is not 0, valid input must be passed. When valid data is passed, it must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • constExpertVOptional can be set to a valid data or null pointer. When constExpertNum is not 0, valid input must be passed. When valid data is passed, it must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [[object Object]moeExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum[object Object]).
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid copy expert IDs must be in the range [[object Object]moeExpertNum + zeroExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum + copyExpertNum[object Object]).
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid constant expert IDs must be in the range [[object Object]moeExpertNum + zeroExpertNum + copyExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum[object Object]).
  • You can pass valid data or a null pointer for [object Object]. If you pass a null pointer, the function of recording the communication duration is disabled. If you pass valid data, it must be a 1D tensor with shape (ep_world_size,), and its data type and data format must be int64 and ND, respectively.[object Object]
  [object Object]
  • [object Object] is not supported in the current version. Pass a null pointer.
  • The shape of [object Object] is (epWorldSize max(tpWorldSize, 1) localExpertNum, ).
  • The TP communicator is not supported.
  • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • sharedExpertXOptional must be a 2D or 3D tensor. (If it is a 2D tensor, the shape is (BS, H). If it is a 3D tensor, the product of the first two dimensions is equal to BS, and the third dimension is equal to H.) This parameter can be passed or not. If it is passed, sharedExpertRankNum must be 0.
  • The value of [object Object] must be in the range [2, 768].
  • The value of [object Object] must be in the range (0, 1024].
  • [object Object] is not supported in the current version. Pass an empty string.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
  • The value of [object Object] must be in the range [0, 4].
  • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
  • The value of [object Object] is 0 or 2 (0 indicates no quantization and 2 indicates int8 quantization). The value 2 is supported only when tpWorldSize < 2.
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.
  • elasticInfoOptional is a reserved parameter and is not supported in the current version. You can pass a null pointer.
  • constExpertAlpha1Optional can be set to a valid data or null pointer. When constExpertNum is not 0, a valid input must be passed. When valid data is passed, the data must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • constExpertAlpha2Optional: You can pass a valid data or null pointer. When constExpertNum is not 0, valid input must be passed. When valid data is passed, it must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • constExpertVOptional: You can pass a valid data or null pointer. When constExpertNum is not 0, valid input must be passed. When valid data is passed, it must be a 2D tensor with the shape of [object Object](constExpertNum, H)[object Object]. The data type must be the same as that of expandX.
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [[object Object]moeExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum[object Object]).
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid copy expert IDs must be in the range [[object Object]moeExpertNum + zeroExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum + copyExpertNum[object Object]).
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid constant expert IDs must be in the range [[object Object]moeExpertNum + zeroExpertNum + copyExpertNum[object Object], [object Object]moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum[object Object]).
  • [object Object] is a reserved parameter, which is not supported in the current version. Pass a null pointer.[object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown:

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic Computation

  • [object Object] defaults to a deterministic implementation.

• Driver constraints:

  • The driver versions of all nodes in the operator communicator must be the same.

• API constraints::

  • [object Object] and [object Object] must be used together. The [object Object], [object Object], [object Object], and [object Object] outputs of [object Object] must be directly passed to the corresponding parameters of [object Object]. The service logic cannot depend on the specific values of these tensors.

• Parameter consistency constraints:

  • The groupEp, epWorldSize, moeExpertNum, groupTp, tpWorldSize, expertShardType, sharedExpertNum, sharedExpertRankNum, globalBS, commAlg, and HCCL_BUFFSIZE parameters of all devices must be the same and consistent with those of aclnnMoeDistributeDispatchV4.

• Product constraints:

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, the "rank" in the parameter description indicates a single die.

• Shape variable constraints:

  [object Object]

• Environment variables constraints:

  • HCCL_BUFFSIZE: Before calling this API, check whether the value of the HCCL_BUFFSIZE environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
      • If [object Object] is set to [object Object] or [object Object], select the [object Object] or [object Object] formula based on the HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE environment variables.
      • If commAlg is set to fullmesh, the value must be [object Object]2 * (BS * epWorldSize * min(localExpertNum, K) * H * sizeof(uint16) + 2MB)[object Object].
      • If commAlg is set to hierarchy, the value must be greater than or equal to (moeExpertNum + epWorldSize / 4) * Align512(maxBS * (H * 2 + 16 * Align8(K))) * 1B + 8MB, where Align8(x) = ((x + 8 - 1) / 8) 8 and Align512(x) = ((x + 512 - 1) / 512) 512.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:
      • If commAlg is set to "" or is not set, the value must satisfy [object Object]>= 2 and >= 2 * (localExpertNum * maxBS * epWorldSize * Align512(Align32(2 * H) + 44) + (K + sharedExpertNum) * maxBS * Align512(2 * H))[object Object]. In addition, [object Object]localExpertNum[object Object] must be set to the number of experts on the MoE expert card, where [object Object]Align512(x) = ((x + 512 - 1) / 512) * 512, Align32(x) = ((x + 32 - 1) / 32) * 32[object Object].
      • If commAlg is set to "hierarchy", the value must satisfy [object Object](moeExpertNum * maxBS * (H * 2 + (3 * (K + 7) / 8 * 8)) * 4 + 64) + 404 * 1024 * 1024[object Object].
  • HCCL_INTRA_PCIE_ENABLE/HCCL_INTRA_ROCE_ENABLE:
    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This environment variable is not recommended. You are advised to set commAlg to [object Object].
  • HCCL_LOGIC_SUPERPOD_ID
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: When commAlg is set to "hierarchy", this environment variable needs to be configured based on the different superpods. For example, set the environment variable to export HCCL_LOGIC_SUPERPOD_ID=0 and export HCCL_LOGIC_SUPERPOD_ID=1 for two devices respectively.

• Constraints on the use of communication domains:

  • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
  • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: The communication domain supports cross-SuperPoD deployment only when commAlg is set to "hierarchy". In other scenarios, nodes in a communication domain must reside within the same SuperPoD.

• Networking constraints:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

• Other constraints:

  • In the formulas, [object Object] denotes integer division.
  • [object Object]moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum < MAX_INT32[object Object]

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]

  This example supports running A2 operators on a single-node system with 2 to 8 ranks. You can set EP_WORLD_SIZE_A2 to the number of ranks and change the value of moeExpertNum in the sample code to ensure that moeExpertNum can be exactly divided by EP_WORLD_SIZE_A2.

  • Operator build: The following is the operator build command. Both the moe_distribute_dispatch_v2 and moe_distribute_combine_v2 operators need to be built and executed in pairs.

    [object Object]

  • Create the sample code for the [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]. After the compilation is complete, create a test file test_aclnn_moe_distribute_combine_v4.cpp using the Atlas A2 sample code in the operator directory by referring to the existing file.

  • Run the operator sample. The following command will execute all sample code files in the operator directory.

    [object Object]

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object] sample code:

    [object Object]

• Ascend 950PR/Ascend 950DT: Refer to the preparation section and sample code in the API call example. Set the involved variables again according to the preceding restrictions. For the scenario parameters added in the V4 API compared with the V3 API, pass the values according to the preceding parameter description.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  For details, see .

• The sample code is as follows:

  [object Object]