[object Object]

[object Object][object Object]undefined[object Object]

• Description: Quantizes token data (optional). When there is TP domain communication, AllToAllV communication in the EP domain is performed first, and then AllGatherV TP domain communication is performed. When there is no such communication, AllToAllV communication in the EP domain is performed.

  Compared with the [object Object] API, this API has the following changes:

  • Special expert scenarios are supported:

    • [object Object]: Enabled by setting the [object Object] parameter to a value greater than 0.

      $$Moe(oriXOptional) = 0$$

    • [object Object]: Enabled by setting the [object Object] parameter to a value greater than 0 and setting a valid value for the [object Object] parameter.

      $$Moe(oriXOptional) = oriXOptional$$

    • [object Object]: Enabled by setting the [object Object] parameter to a value greater than 0 and setting valid values for the [object Object], [object Object], [object Object], and [object Object] parameters.

      $$Moe(oriXOptional) = constExpertAlpha1Optional * oriXOptional + constExpertAlpha2Optional * constExpertVOptional$$

    For details, see the following parameter description.

    For details about the [object Object], [object Object], [object Object], and [object Object] parameters, see the [object Object] file.

• Formula:

  • Case 1: If there is no communication in the TP domain,

    $$expandXOut = AllToAllV(X)\\$$

  • Case 2: If there is communication in the TP domain,

    $$allToAllOut = AllToAllV(X)\\ expandXOut = AllGatherV(allToAllOut)\\$$

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This API must be used together with [object Object].

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: This API must be used together with [object Object] or [object Object].

[object Object]

[object Object]

Each operator has calls. First, [object Object] is called to obtain the workspace size required for computation and the executor that contains the operator computation process. Then, [object Object] is called to perform computation.

[object Object]

• Parameters

  [object Object][object Object]
  • The value of [object Object] can be [object Object], [object Object], [object Object], or [object Object]. It is recommended to use [object Object] with driver version 25.0.RC1.1 or later. When set to [object Object] or [object Object], the communication algorithm is selected based on HCCL environment variables (not recommended). [object Object] indicates that tokens are directly transmitted through RDMA. [object Object] indicates a two-stage communication process: intra-server communication followed by inter-server communication, which reduces cross-server data transmission.
  • When commAlg is set to "hierarchy" or HCCL_INTRA_PCIE_ENABLE is set to 1 and HCCL_INTRA_ROCE_ENABLE is set to 0, nullptr must be passed to scalesOptional. When commAlg is set to "fullmesh", valid data or null pointer can be passed to scalesOptional.
  • The value of xActiveMaskOptional depends on the value of commAlg. For "fullmesh", xActiveMaskOptional must be a 1D tensor with the shape of (BS,). true must be placed before false. For example, {true, false, true} is invalid. For "hierarchy", the current version does not support this parameter. You can pass a null pointer.
  • The value of expertScalesOptional must be a 2D tensor with the shape of (BS, K).
  • The value of epWorldSize depends on the value of commAlg. For "fullmesh", the value can be 2, 3, 4, 5, 6, 7, 8, 16, 32, 64, 128, 192, 256, or 384. For "hierarchy", the value can be 16, 32, or 64.
  • The value of moeExpertNum ranges from (0, 512].
  • [object Object] is not supported in the current version. Pass an empty string.
  • The current version does not support [object Object], [object Object], [object Object], [object Object], and [object Object]. Pass 0 for these parameters.
  • The shape of epRecvCountsOut is (moeExpertNum + 2 globalBS K * serverNum,). (The first moeExpertNum elements indicate the number of received tokens, and the remaining elements indicate the information related to reduce before communication.)
  • Currently, TP domain communication is not supported.
  • [object Object] must be a 1D tensor with shape (A ).
  • [object Object] supports 0 (non-quantization) and 2 (dynamic quantization).
  • [object Object] is not supported in the current version. Pass a null pointer.
  • When commAlg is [object Object], the value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [[object Object], [object Object]).
  • When commAlg is [object Object], the value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid copy expert IDs must be in the range [[object Object], [object Object]).
  • [object Object] is not supported in the current version. Pass 0.[object Object]
  [object Object]
  • [object Object] supports three input modes: "", "fullmesh_v1", and "fullmesh_v2". "": The fullmesh_v1 template is enabled by default. "fullmesh_v1": The fullmesh_v1 template is enabled. "fullmesh_v2": The fullmesh_v2 template is enabled. In this case, commAlg supports only the scenario where tpWorldSize is 1.
  • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
  • [object Object] is not supported in the current version. Pass a null pointer.
  • The value of [object Object] must be in the range [2, 768].
  • The value of [object Object] must be in the range (0, 1024].
  • The value of groupTp is a string of [0, 128) characters and cannot be the same as that of groupEp. This parameter can be left empty only when there is no communication in the TP domain.
  • The value of [object Object] must be in the range [0, 2]. 0 and 1 indicate no TP domain communication. 2 is required when TP domain communication is used.
  • The value of [object Object] must be in the range [0, 1]. [object Object] of each rank in the same TP domain must be unique. If TP domain communication is not used, pass 0.
  • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
  • The value of [object Object] must be in the range [0, 4].
  • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
  • The shape of epRecvCountsOut is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
  • When there is communication in the TP domain, tpRecvCountsOut is a 1D shape tensor, and the shape is (tpWorldSize,).
  • [object Object] is not supported in the current version.
  • [object Object] supports 0 (non-quantization) and 2 (dynamic quantization).
  • [object Object] is not supported in the current version. Pass a null pointer.
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [object Object][moeExpertNum, moeExpertNum + zeroExpertNum)[object Object].
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid expert IDs must be in the range [object Object][moeExpertNum + zeroExpertNum, moeExpertNum + zeroExpertNum + copyExpertNum)[object Object].
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid expert IDs must be in the range [object Object].[object Object]
  [object Object]
  • [object Object] supports three input modes: "", "fullmesh_v1", and "fullmesh_v2". "": The fullmesh_v1 template is enabled by default. "fullmesh_v1": The fullmesh_v1 template is enabled. "fullmesh_v2": The fullmesh_v2 template is enabled. In this case, commAlg supports only the scenario where tpWorldSize is 1.
  • [object Object] must be a 1D tensor with shape (BS, ) or a 2D tensor with shape (BS, K). If it is a 1D tensor, [object Object] must be placed before [object Object]. If it is a 2D tensor and the K values corresponding to tokens are all [object Object], the tokens do not participate in communication.
  • [object Object] is not supported in the current version. Pass a null pointer.
  • The value of [object Object] must be in the range [2, 768].
  • The value of [object Object] must be in the range (0, 1024].
  • [object Object] is not supported in the current version. Pass an empty string.
  • [object Object] is not supported in the current version. Pass 0.
  • [object Object] is not supported in the current version. Pass 0.
  • The value of [object Object] must be 0, indicating that shared expert ranks are placed in front of MoE expert ranks.
  • The value of [object Object] must be in the range [0, 4].
  • The value of [object Object] must be in the range [0, epWorldSize). If the value is 0, [object Object] is 0 or 1. If the value is not 0, [object Object] is 0.
  • The shape of epRecvCountsOut is (epWorldSize max(tpWorldSize, 1) localExpertNum,).
  • When there is communication in the TP domain, tpRecvCountsOut is a 1D shape tensor, and the shape is (tpWorldSize,).
  • [object Object] is not supported in the current version.
  • quantMode supports 0 (non-quantization), 1 (static quantization), 2 (per-token dynamic quantization), 3 (per-group dynamic quantization), and 4 (mxfp8 dynamic quantization).
  • [object Object] is not supported in the current version. Pass a null pointer.
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid zero expert IDs must be in the range [object Object][moeExpertNum, moeExpertNum + zeroExpertNum)[object Object].
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid expert IDs must be in the range [object Object][moeExpertNum + zeroExpertNum, moeExpertNum + zeroExpertNum + copyExpertNum)[object Object].
  • The value of [object Object] must be in the range [0, MAX_INT32), where MAX_INT32 = 2^31 - 1. Valid expert IDs must be in the range [object Object].[object Object]

• Returns

  [object Object]: status code. For details, see .

  The first-phase API implements input parameter verification. The following errors may be thrown.

  [object Object]

[object Object]

• Parameters

  [object Object]

• Returns

  [object Object]: status code. For details, see .

[object Object]

• Deterministic Computation

  • [object Object] defaults to a deterministic implementation.

• Driver restrictions:

  • The driver versions of all nodes in the operator communicator must be the same.

• API constraints:

  • [object Object] and CombineV3 operators must be used together. The [object Object], [object Object], [object Object], and [object Object] outputs of [object Object] must be directly passed to the corresponding parameters of [object Object]. The service logic cannot depend on the specific values of these tensors.

• Parameter consistency constraints:

  • The values of the groupEp, epWorldSize, moeExpertNum, groupTp, tpWorldSize, expertShardType, sharedExpertNum, sharedExpertRankNum, globalBS, commAlg, and HCCL_BUFFSIZE parameters of all devices must be the same and consistent with those of the corresponding parameters of the CombineV3 series operators.

• Product constraints:

  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: In this scenario, a single rank contains dual dies. Therefore, the "rank" in the parameter description indicates a single die.

• Shape variable constraints:

  [object Object]

• Environment variables constraints:

  • HCCL_BUFFSIZE:

    Before calling this API, check whether the value of the [object Object] environment variable is proper. This environment variable indicates the buffer size occupied by a single communication domain, in MB. If this environment variable is not set, the default value 200 MB is used.

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:
      • If [object Object] is set to [object Object] or [object Object], select the [object Object] or [object Object] formula based on the [object Object] and [object Object] environment variables.
      • If commAlg is set to "fullmesh", the size must be greater than or equal to 2 x (BS x epWorldSize x min(localExpertNum, K) x H x sizeof(uint16) + 2MB).
      • If commAlg is set to "hierarchy", the size must be >= (moeExpertNum + epWorldSize / 4) * Align512(maxBS * (H * 2 + 16 * Align8(K))) * 1B + 8MB, where Align8(x) = ((x + 8 - 1) / 8) x 8 and Align512(x) = ((x + 512 - 1) / 512) x 512.
    • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object] /Ascend 950PR/Ascend 950DT:
      • In the ep communicator, if commAlg is set to "fullmesh_v1", an empty string, or a null pointer, the size must be greater than or equal to ≥ 2 * (localExpertNum * maxBS * epWorldSize * Align512(Align32(2 * H) + 64) + (K + sharedExpertNum) * maxBS * Align512(2 * H)).
      • In the ep communicator, if commAlg is set to "fullmesh_v2", the size must be greater than or equal to ≥ 2 * (localExpertNum * maxBS * epWorldSize * 480Align512(Align32(2 * H) + 64) + (K + sharedExpertNum) * maxBS * Align512(2 * H)).
      • In a TP communication domain, the value must satisfy ≥ (A × Align512(Align32(h × 2) + 44) + A × Align512(h × 2)) × 2.
      • 480Align512(x) = ((x + 480 - 1) / 480) × 512, Align512(x) = ((x + 512 - 1) / 512) × 512 and Align32(x) = ((x + 32 - 1) / 32) × 32.

  • HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE:

    • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: This environment variable is not recommended. You are advised to set commAlg to [object Object].

• Constraints on the use of the communication domains:

  • [object Object] and [object Object] in a model support only the same EP communication domain, and no other operators are allowed in the communication domain.
  • [object Object] and [object Object] in a model support only the same TP communication domain or both do not support a TP communication domain. If a TP communication domain is supported, no other operators are allowed in the communication domain.
  • [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]: Nodes in a communication domain must be in the same SuperPoD. Cross-SuperPoD nodes are not supported.

• Networking constraints:

  • [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]: In multi-server scenarios, only switch-based networking is supported, and direct point-to-point networking between two servers is not supported.

• Other constraints:

  • In the formulas, / denotes integer division.
  • [object Object]moeExpertNum + zeroExpertNum + copyExpertNum + constExpertNum < MAX_INT32[object Object]

[object Object]

• [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

  In this example, the A2 operator can run in a single-node environment with 2 to 8 devices. You can set EP_WORLD_SIZE_A2 to the number of devices and change the value of moeExpertNum in the sample code so that moeExpertNum can be exactly divided by EP_WORLD_SIZE_A2.

  • Operator compilation: The operator compilation commands are as follows. Both the moe_distribute_dispatch_v2 and moe_distribute_combine_v2 operators need to be compiled. These two operators must be executed in pairs.

    [object Object]

  • Create the sample code for the [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]. After the compilation is complete, create a test file test_aclnn_moe_distribute_dispatch_v3.cpp using the A2 sample code in the directory of the operator by referring to the existing file.

  • Run the operator sample. The command for running the operator sample is as follows. This command will execute all sample code files in the directory of the operator.

    [object Object]

  • Sample code for the [object Object]Atlas A2 training products/Atlas A2 inference products[object Object]:

    [object Object]

• Ascend 950PR/Ascend 950DT: Refer to the preparation section and sample code in the API call example, and reset the involved variables according to the preceding restrictions. For the scenario parameters added in the V3 API compared with the V2 API, pass the values according to the preceding parameter description.

• [object Object]Atlas A3 training products/Atlas A3 inference products[object Object]:

  For details, see .

• The sample code is as follows:

  [object Object]