aclnnMoeDistributeCombine

📄 View Source Code

Applicable Products

Function

• Description: When TP domain communication exists, first perform ReduceScatterV communication, then perform AllToAllV communication, and finally combine the received data (multiply by weights and sum). When TP domain communication does not exist, perform AllToAllV communication, and finally combine the received data (multiply by weights and sum).

• Formula:

  • When there is no TP domain communication:
  $$ataOut = AllToAllV(expandX)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$
  • When TP-domain communication exists:
  $$rsOut = ReduceScatterV(expandX)\\ ataOut = AllToAllV(rsOut)\\ xOut = Sum(expertScales * ataOut + expertScales * sharedExpertX)$$

Note: This API must be used together with aclnnMoeDistributeDispatch. It is equivalent to returning data along the original path used by the MoeDistributeDispatch operator to collect data.

Function Prototype

Each operator is divided into a two-phase API. You must first call the aclnnMoeDistributeCombineGetWorkspaceSize API to obtain the required workspace size and the executor that contains the operator computation process, and then call the aclnnMoeDistributeCombine API to perform the computation.

aclnnStatus aclnnMoeDistributeCombineGetWorkspaceSize(
    const aclTensor* expandX,
    const aclTensor* expertIds,
    const aclTensor* expandIdx,
    const aclTensor* epSendCounts,
    const aclTensor* expertScales,
    const aclTensor* tpSendCounts,
    const aclTensor* xActiveMask,
    const aclTensor* activationScale,
    const aclTensor* weightScale,
    const aclTensor* groupList,
    const aclTensor* expandScales,
    const char*      groupEp,
    int64_t          epWorldSize,
    int64_t          epRankId,
    int64_t          moeExpertNum,
    const char*      groupTp,
    int64_t          tpWorldSize,
    int64_t          tpRankId,
    int64_t          expertShardType,
    int64_t          sharedExpertNum,
    int64_t          sharedExpertRankNum,
    int64_t          globalBS,
    int64_t          outDtype,
    int64_t          commQuantMode,
    int64_t          groupListType,
    aclTensor*       x,
    uint64_t*        workspaceSize,
    aclOpExecutor**  executor)

aclnnStatus aclnnMoeDistributeCombine(
    void            *workspace,
    uint64_t        workspaceSize,
    aclOpExecutor   *executor,
    aclrtStream     stream)

aclnnMoeDistributeCombineGetWorkspaceSize

• Parameters

  Parameter	Input/Output	Description	Instruction	Data Type	Data Format	Dimension (Shape)	Non-contiguous Tensor
  expandX	Input	Token features expanded based on expertIds.	Must be a 2D tensor.	FLOAT16, BFLOAT16	ND	(max(tpWorldSize, 1) * A , H)	√
  expertIds	Input	Indices of the topK experts for each token.	Must be a 2D tensor.	INT32	ND	(BS, K)	√
  expandIdx	Input	Corresponds to the expandIdx output in aclnnMoeDistributeDispatch.	Must be a 1D tensor.	INT32	ND	(BS*K, )	√
  epSendCounts	Input	Corresponds to the epRecvCounts output in aclnnMoeDistributeDispatch.	Must be a 1D tensor.	INT32	ND	-	√
  expertScales	Input	Weights of the topK experts for each token.	Must be a 2D tensor.	FLOAT32	ND	(BS, K)	√
  tpSendCounts	Input	Corresponds to the tpRecvCounts output in aclnnMoeDistributeDispatch.	Pass the parameter when TP domain communication is required; pass a null pointer when no TP domain communication is needed.	INT32	ND	-	-
  xActiveMask	Input	Reserved parameter.	Not supported in the current version. Pass a null pointer.	-	ND	-	-
  activationScale	Input	Reserved parameter.	Not supported in the current version. Pass a null pointer.	-	ND	-	-
  weightScale	Input	Reserved parameter.	Not supported in the current version. Pass a null pointer.	-	ND	-	-
  groupList	Input	Reserved parameter.	Not supported in the current version. Pass a null pointer.	-	ND	-	-
  expandScales	Input	Corresponds to the expandScales output in aclnnMoeDistributeDispatch.	-	FLOAT32	ND	-	√
  groupEp	Input	Name of the EP communication domain (expert parallel communication domain).	The string length ranges from [1, 128) and cannot be the same as groupTp.	STRING	-	-	-
  epWorldSize	Input	Size of the EP communication domain.	-	INT64	-	-	-
  epRankId	Input	ID of the local card in the EP domain.	Value range: [0, epWorldSize). The epRankId of each card in the same EP communication domain must be unique.	INT64	-	-	-
  moeExpertNum	Input	Number of MoE experts.	Value range: (0, 512], and moeExpertNum % (epWorldSize - sharedExpertRankNum) = 0 must be satisfied.	INT64	-	-	-
  groupTp	Input	Name of the TP communication domain (data parallel communication domain).	Must not be the same as groupEp.	STRING	-	-	-
  tpWorldSize	Input	TP communication domain size.	-	INT64	-	-	-
  tpRankId	Input	ID of the current card in the TP domain.	The tpRankId of each card in the same TP communication domain is unique.	INT64	-	-	-
  expertShardType	Input	Indicates the distribution type of shared expert cards.	-	INT64	-	-	-
  sharedExpertNum	Input	Indicates the number of shared experts (a shared expert can be replicated and deployed across multiple cards).	-	INT64	-	-	-
  sharedExpertRankNum	Input	Indicates the number of shared expert cards.	-	INT64	-	-	-
  globalBS	Input	Global batch size in the EP domain.	-	INT64	-	-	-
  outDtype	Input	Specifies the data type of the output x. This is a reserved parameter and is not supported in the current version. Pass 0.	-	INT64	-	-	-
  commQuantMode	Input	Communication quantization type.	-	INT64	-	-	-
  groupListType	Input	Format of the group list.	Reserved parameter. Not supported in the current version. Pass 0.	INT64	-	-	-
  x	Output	Indicates the processed tokens.	Must be a 2D tensor. The data type and data format must be the same as those of expandX.	FLOAT16, BFLOAT16	ND	(BS, H)	-
  workspaceSize	Output	Returns the workspace size to be applied for on the Device side.	-	UINT64	-	-	-
  executor	Output	Returns the operator executor, which contains the computation flow of the operator.	-	aclOpExecutor*	-	-	-

  • Atlas A2 training products/Atlas A2 inference products:

    • The shared expert scenario is not supported.

    • The shape of epSendCounts is (moeExpertNum + 2 * globalBS * K * serverNum, ), where K indicates the number of topK experts. The first moeExpertNum values indicate the number of tokens received from each card in the EP communication domain, and the last 2 * globalBS * K * serverNum values are used to store the number of tokens and communication domain offsets that can be reduced in advance by combine before inter-server/intra-server communication. When globalBS = 0, the calculation is based on BS * epWorldSize.

    • TP domain communication is not supported.

    • expandScales must be a 1D tensor with the shape (A, ).

    • epWorldSize supports the values 16, 32, and 64.

    • moeExpertNum must also satisfy moeExpertNum / (epWorldSize - sharedExpertRankNum) <= 24.

    • groupTp is not supported in the current version. Pass an empty string.

    • tpWorldSize, tpRankId, expertShardType, sharedExpertNum, and sharedExpertRankNum are not supported in the current version. Pass 0 for each.

    • When the BS is consistent across ranks, globalBS = BS epWorldSize or 0. When the BS is inconsistent across ranks, globalBS = maxBS epWorldSize or 256 epWorldSize (maxBS is the maximum BS of a single rank. It is recommended to pass maxBS epWorldSize).

    • commQuantMode can be 0 or 2. 0 indicates no communication quantization, and 2 indicates int8 communication quantization (2 is supported only when HCCL_INTRA_PCIE_ENABLE=1, HCCL_INTRA_ROCE_ENABLE=0, and the driver version is ≥ 25.0.RC1.1).

  • Atlas A3 training products/Atlas A3 inference products:

    • The shape of epSendCounts is (epWorldSize * max(tpWorldSize, 1) * localExpertNum, ).

    • When TP domain communication exists, tpSendCounts is a 1D tensor with the shape (tpWorldSize, ).

    • expandScales is a reserved parameter and is not supported in the current version. Pass a null pointer.

    • epWorldSize supports the following values: 8, 16, 32, 64, 128, 144, 256, and 288.

    • The string length of groupTp ranges from 0 to 128 (exclusive). It cannot be the same as groupEp. Passing an empty string is supported only when no TP domain communication exists.

    • tpWorldSize: The value range is [0, 2]. 0 and 1 indicate no TP domain communication. When TP domain communication exists, only 2 is supported.

    • tpRankId: The value range is [0, 1]. The tpRankId of each card in the same TP communication domain must be unique. When there is no TP domain communication, pass 0.

    • expertShardType: Currently, only 0 is supported, indicating that shared expert cards are placed before MoE expert cards.

    • sharedExpertNum: The current value range is [0, 1]. 0 indicates no shared expert, and 1 indicates one shared expert. Currently, only 1 is supported in this version.

    • sharedExpertRankNum: The current value range is [0, epWorldSize). When the value is not 0, epWorldSize % sharedExpertRankNum = 0 must be satisfied.

    • When the BS is consistent across all ranks, globalBS = BS * epWorldSize or 0; when the BS is inconsistent across ranks, globalBS = maxBS * epWorldSize (maxBS is the maximum BS value on a single card).

    • commQuantMode has a value range of 0 or 2. 0 indicates no quantization for communication, and 2 indicates int8 quantization for communication.

  • Ascend 950PR/Ascend 950DT:

    • The shape of epSendCounts is (epWorldSize * max(tpWorldSize, 1) * localExpertNum, ).

    • TP domain communication is not supported currently.

    • expandScales is a reserved parameter and is not supported in the current version. Pass a null pointer.

    • epWorldSize supports the following values: 2, 4, 8, 16, 32, 64, 128, 144, 256, and 288.

    • groupTp is not supported in the current version. Pass an empty string.

    • tpWorldSize is not supported in the current version. Pass 1.

    • tpRankId is not supported in the current version. Pass 0.

    • expertShardType: Currently, only 0 is supported, indicating that shared expert cards are placed before MoE expert cards.

    • sharedExpertNum: The current value range is [0, 1]. 0 indicates no shared expert, and 1 indicates one shared expert. Currently, only 1 is supported.

    • sharedExpertRankNum: The current value range is [0, epWorldSize). When it is not 0, epWorldSize % sharedExpertRankNum = 0 must be satisfied.

    • When BS is consistent across ranks, globalBS = BS epWorldSize or 0. When BS is inconsistent across ranks, globalBS = maxBS epWorldSize (maxBS is the maximum BS value on a single card).

    • commQuantMode: Currently, only 0 is supported. 0 indicates that communication is not quantized.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

  The first-phase API performs input parameter validation and returns an error in the following scenarios:

  Return Value	Error Code	Description
  ACLNN_ERR_PARAM_NULLPTR	161001	The required input and output tensor parameters are null pointers.
  ACLNN_ERR_PARAM_INVALID	161002	The data types of the input and output are not within the supported range.
  ACLNN_ERR_INNER_TILING_ERROR	561002	The shape of the input or output is not within the supported range.
  		The parameter value is not within the supported range.

aclnnMoeDistributeCombine

• Parameters

  Parameter	Input/Output	Description
  workspace	Input	Workspace memory address applied on the device side.
  workspaceSize	Input	Workspace size applied on the device side, obtained by the first-phase API aclnnMoeDistributeCombineGetWorkspaceSize.
  Executor	Input	Op executor, which contains the operator computation process.
  Stream	Input	Specifies the stream for task execution.

• Return Value

  aclnnStatus: return code. For details, see aclnn Return Code.

Constraints

1. Deterministic computation:

   • aclnnMoeDistributeCombine defaults to a deterministic implementation.

2. Driver constraints:

   • The driver versions of all nodes in the operator communication domain must be the same.

3. The aclnnMoeDistributeDispatch API and the aclnnMoeDistributeCombine API must be used together. For details, see the example.

4. Across different product models, communication algorithms, or versions, the element values in the expandIdx, epRecvCounts, tpRecvCounts, and expandScales tensor outputs of aclnnMoeDistributeDispatch may differ. When using them, directly pass the preceding tensors to the corresponding parameters of aclnnMoeDistributeCombine. Other model business logic must not depend on them.

5. The values of the groupEp, epWorldSize, moeExpertNum, groupTp, tpWorldSize, expertShardType, sharedExpertNum, sharedExpertRankNum, and globalBs parameters used during the API call must be consistent across all cards, must also be consistent across different layers in the network, and must be consistent with the corresponding parameters of aclnnMoeDistributeDispatch.

6. Atlas A3 training products/Atlas A3 inference products: In this scenario, a single card contains dual DIEs (referred to as "dies"). Therefore, "this card" in the parameter description refers to a single DIE.

7. Description of the shape format in the parameter description:

   • A: Indicates the maximum number of tokens to be distributed by the current card. The value range is as follows:

     • For shared experts, the condition (A = BS * epWorldSize * sharedExpertNum / sharedExpertRankNum) must be satisfied.

     • For MoE experts, when globalBS is 0, the condition (A >= BS * epWorldSize * min(localExpertNum, K)) must be satisfied; when globalBS is non-zero, the condition (A >= globalBS * min(localExpertNum, K)) must be satisfied.

   • H: Indicates the hidden size:

     • Atlas A2 training products/Atlas A2 inference products: The value range is (0, 7168] and must be an integer multiple of 32.

     • Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT: The value is 7168.

   • BS: Indicates the batch sequence size (the number of tokens finally output by the current card):

     • Atlas A2 training products/Atlas A2 inference products: The value range is (0 < BS ≤ 256).

     • Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT: The value range is (0 < BS ≤ 512).

   • K: Indicates selecting the top K experts, and must satisfy (0 < K ≤ moeExpertNum):

     • Atlas A2 training products/Atlas A2 inference products: The value range is (0 < K ≤ 16).

     • Atlas A3 training products/Atlas A3 inference products, Ascend 950PR/Ascend 950DT: The value range is (0 < K ≤ 8).

   • serverNum: Indicates the number of server nodes. Only 2, 4, and 8 are supported.

   • localExpertNum: Indicates the number of experts on the current card:

     • For a shared expert card, (localExpertNum = 1).

     • For MoE expert cards, (localExpertNum = moeExpertNum / (epWorldSize - sharedExpertRankNum)); when (localExpertNum > 1), TP domain communication is not supported.

8. HCCL_BUFFSIZE:

   Before calling this API, check whether the value of the HCCL_BUFFSIZE environment variable is appropriate. This environment variable specifies the memory size occupied by a single communication domain, in MB. The default value is 200 MB when not configured:

   • Atlas A2 training products/Atlas A2 inference products:

     • The configured size must satisfy (≥ 2 * (BS * epWorldSize * min(localExpertNum, K) * H * sizeof(uint16) + 2 MB)).

   • Atlas A3 training products/Atlas A3 inference products:

     • Within the ep communication domain: Set the size requirement (≥ 2) and ensure that (1024^2 (HCCL_BUFFSIZE - 2) / 2 ≥ BS 2 (H + 128) (epWorldSize * localExpertNum + K + 1)), where localExpertNum uses the local expert count per card of the MoE expert card.

     • Within the tp communication domain: Set the size requirement ≥ A (H 2 + 128) * 2.

   • Ascend 950PR/Ascend 950DT: Set the size requirement (≥ 2) and ensure that (1024^2 (HCCL_BUFFSIZE - 2) / 2 ≥ BS 2 (H + 128) (epWorldSize * localExpertNum + K + 1)), where localExpertNum uses the local expert count per card of the MoE expert card.

9. HCCL_INTRA_PCIE_ENABLE and HCCL_INTRA_ROCE_ENABLE:

   Atlas A2 training products/Atlas A2 inference products: Set the environment variables HCCL_INTRA_PCIE_ENABLE = 1 and HCCL_INTRA_ROCE_ENABLE = 0 to reduce the amount of cross-machine communication data, which may improve operator performance. In this case, HCCL_BUFFSIZE must satisfy (≥ moeExpertNum * BS * (H * sizeof(dtypeX) + 4 * ((K + 7) / 8 * 8) * sizeof(uint32)) + 4MB + 100MB); and for the input parameter moeExpertNum, only (moeExpertNum % (epWorldSize - sharedExpertRankNum) = 0) is required, while (moeExpertNum / (epWorldSize - sharedExpertRankNum) ≤ 24) is not required.

10. The "/" in the formulas in this document indicates integer division.

11. Communication domain usage constraints:

    • aclnnMoeDistributeCombine and aclnnMoeDistributeDispatch in a model support only the same EP communication domain, and no other operators are allowed in this communication domain.

    • aclnnMoeDistributeCombine and aclnnMoeDistributeDispatch in a model support only the same TP communication domain or neither supports a TP communication domain. When a TP communication domain exists, no other operators are allowed in this communication domain.

    • Atlas A3 training products/Atlas A3 inference products: Nodes within a communication domain must reside within a single super node. Cross-super-node communication is not supported.

12. Networking constraints:

    • Atlas A2 training products/Atlas A2 inference products: In multi-machine scenarios, only switch networking is supported. Direct dual-machine networking is not supported.

Example

• Atlas A2 training products/Atlas A2 inference products:

  • File preparation:

    1. Create and modify the rank_table_m2.json file as instructed below.

    2. Copy the project to both servers, and configure the rank_table_m2.json file content based on the device IP of each machine. Ensure that the rank_table_m2.json files on both machines remain identical.

    3. Install the CANN package, and compile and run the sample as instructed below.

  • About rankTable:

    1. Developers can configure the NPU resource information for collective communication through the ranktable file. For detailed configuration, see "Communication Function Development > Cluster Information Configuration > Configuring Resource Information in the ranktable File" in Collective Communication User Guide.

    2. Use cat /etc/hccn.conf or for i in seq 0 7; do echo "===================> dev$i, NPU$((i+1))"; hccn_tool -i $i -ip -g; done to query the device IP of the machine. Then fill in the JSON file by referring to the collective communication documentation.

    Note: In a two-machine 16-card scenario, the device_id of both machines is 0–7, the rank_id of one machine is 0–7, and the rank_id of the other machine is 8–15. In a single-machine 16-card scenario, both device_id and rank_id are 0–15.

  • Environment variable configuration:

    # Set three environment variables before running.
    ## FIRST_RANK_ID description: In a two-machine 16-card scenario, set it to 0 on one machine and 8 on the other.
    ## # For example, export FIRST_RANK_ID=0
    export RANK_TABLE_FILE=/home/path/to/rank_table_m2.json
    export FIRST_RANK_ID=<starting rank_id of the device>
    ## # ENV_DEV_NUM description: Set this variable based on the number of cards on the current machine. For example, in a two-machine 16-card scenario, set the two machines to 16.
    export ENV_DEV_NUM=16

  • Machine count setting:

    In a two-machine 16-card scenario, set the MACHINE_NUM parameter to 2, that is,

    const uint32_t MACHINE_NUM = 2;

    No modification is required for a single-machine 16-card scenario.

  • Operator compilation and execution:

    Compile the operators on all machines. The operator compilation command is as follows. Both moe_distribute_dispatch and moe_distribute_combine operators need to be compiled, and these two operators must be executed in pairs:

    bash build.sh --pkg --soc=ascend910b --ops=moe_distribute_dispatch,moe_distribute_combine

    Run the operator example on all machines (in a two-machine scenario, execute the operator simultaneously in the terminals). The execution command is as follows:

    bash build.sh --run_example --ops=moe_distribute_combine eager cust

• Atlas A3 training products/Atlas A3 inference products :

  • Environment Variable Configuration:

Set the environment variable ENV_DEV_NUM before running. No need to configure the ranktable file or the environment variables RANK_TABLE_FILE and FIRST_RANK_ID

ENV_DEV_NUM Description: Set this variable based on the number of NPUs on the current machine. For example, for a single machine with 16 NPUs, set the variable to 16

    export ENV_DEV_NUM=16

    ```

The following provides example code for reference only. For details about compilation and running, see Compile and Run Samples.

• Atlas A2 training products/Atlas A2 inference products , Atlas A3 training products/Atlas A3 inference products , Ascend 950PR/Ascend 950DT:

  
  #include <thread>
  #include <iostream>
  #include <string>
  #include <vector>
  #include <memory>
  #include <cstdio>
  #include "acl/acl.h"
  #include "hccl/hccl.h"
  #include "aclnn/opdev/fp16_t.h"
  #include "aclnnop/aclnn_moe_distribute_dispatch.h"
  #include "aclnnop/aclnn_moe_distribute_combine.h"
  
  #define CHECK_RET(cond, return_expr) \
      do {                             \
          if (!(cond)) {               \
              return_expr;             \
          }                            \
      } while (0)
  
  #define LOG_PRINT(message, ...)         \
      do {                                \
          printf(message, ##__VA_ARGS__); \
      } while(0)
  
  struct Args {
      uint32_t rankId;
      uint32_t epRankId;
      uint32_t tpRankId;
      HcclComm hcclEpComm;
      HcclComm hcclTpComm;
      aclrtStream dispatchStream;
      aclrtStream combineStream;
      aclrtContext context;
  };
  
  const uint32_t MACHINE_NUM = 1;
  const char* rank_table_file = std::getenv("RANK_TABLE_FILE");
  const char* first_rank_id = std::getenv("FIRST_RANK_ID");
  const char* env_dev_num = std::getenv("ENV_DEV_NUM");
  
  uint32_t EP_WORLD_SIZE = 0;
  uint32_t TP_WORLD_SIZE = 0;
  uint32_t DEV_NUM = 0;
  
  int64_t GetShapeSize(const std::vector<int64_t> &shape)
  {
      int64_t shape_size = 1;
      for (auto i : shape) {
          shape_size *= i;
      }
      return shape_size;
  }
  
  template<typename T>
  int CreateAclTensor(const std::vector<T> &hostData, const std::vector<int64_t> &shape, void **deviceAddr,
      aclDataType dataType, aclTensor **tensor)
  {
      auto size = GetShapeSize(shape) * sizeof(T);
      auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtMalloc failed. ret: %d\n", ret); return ret);
      ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtMemcpy failed. ret: %d\n", ret); return ret);
      std::vector<int64_t> strides(shape.size(), 1);
      for (int64_t i = shape.size() - 2; i >= 0; i--) {
          strides[i] = shape[i + 1] * strides[i + 1];
      }
      *tensor = aclCreateTensor(
          shape.data(), shape.size(), dataType, strides.data(), 0,
          aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(), *deviceAddr
      );
      return 0;
  }
  
  int launchOneThreadDispatchAndCombine(Args &args)
  {
      int ret = aclrtSetCurrentContext(args.context);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtSetCurrentContext failed. ret: %d\n", ret); return ret);
  
      char hcomEpName[128] = {0};
      ret = HcclGetCommName(args.hcclEpComm, hcomEpName);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] HcclGetEpCommName failed. ret: %d\n", ret); return -1);
      char hcomTpName[128] = {0};
      if (!rank_table_file && !first_rank_id) {
          ret = HcclGetCommName(args.hcclTpComm, hcomTpName);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] HcclGetTpCommName failed. ret: %d\n", ret); return -1);
      }
      LOG_PRINT(
          "[INFO] rank = %d, hcomEpName = %s, hcomTpName = %s, dispatchStream = %p, combineStream = %p, context = %p\n",
          args.rankId, hcomEpName, hcomTpName, args.dispatchStream, args.combineStream, args.context
      );
  
      // Set the scenario.
      int64_t BS = 8;
      int64_t H = 7168;
      int64_t K = 1;
      int64_t expertShardType = 0;
      int64_t sharedExpertNum = 0;
      int64_t sharedExpertRankNum = 0;
      if (!rank_table_file && !first_rank_id) {
          sharedExpertNum = 1;
          sharedExpertRankNum = 1;
      }
      if (rank_table_file && !first_rank_id) {
          sharedExpertNum = 1;
          sharedExpertRankNum = 0;
      }
      int64_t moeExpertNum = EP_WORLD_SIZE - sharedExpertRankNum;
      int64_t quantMode = 0;
      int64_t globalBS = BS * EP_WORLD_SIZE;
      int64_t expertTokenNumsType = 0;
      int64_t outDtype = 0;
      int64_t commQuantMode = 0;
      int64_t groupListType = 0;
      int64_t localExpertNum;
      int64_t A;
      if (args.epRankId < sharedExpertRankNum) {
          // Shared expert card.
          localExpertNum = 1;
          A = globalBS / sharedExpertRankNum;
      } else {
          // MoE expert card.
          localExpertNum = moeExpertNum / (EP_WORLD_SIZE - sharedExpertRankNum);
          A = globalBS * (localExpertNum < K ? localExpertNum : K);
      }
  
      /* Construct device-side input and output variables based on the current scenario. */
      // Declare device-side input and output variables.
      void *xDeviceAddr = nullptr;
      void *expertIdsDeviceAddr = nullptr;
      void *scalesDeviceAddr = nullptr;
      void *expertScalesDeviceAddr = nullptr;
      void *expandXDeviceAddr = nullptr;
      void *dynamicScalesDeviceAddr = nullptr;
      void *expandIdxDeviceAddr = nullptr;
      void *expertTokenNumsDeviceAddr = nullptr;
      void *epRecvCountsDeviceAddr = nullptr;
      void *tpRecvCountsDeviceAddr = nullptr;
      void *expandScalesDeviceAddr = nullptr;
  
      aclTensor *x = nullptr;
      aclTensor *expertIds = nullptr;
      aclTensor *scales = nullptr;
      aclTensor *expertScales = nullptr;
      aclTensor *expandX = nullptr;
      aclTensor *dynamicScales = nullptr;
      aclTensor *expandIdx = nullptr;
      aclTensor *expertTokenNums = nullptr;
      aclTensor *epRecvCounts = nullptr;
      aclTensor *tpRecvCounts = nullptr;
      aclTensor *expandScales = nullptr;
  
      // Define the dimensions of each variable in the current scenario.
      std::vector<int64_t> xShape{BS, H};
      std::vector<int64_t> expertIdsShape{BS, K};
      std::vector<int64_t> scalesShape{(sharedExpertRankNum > 0) ? 1 + moeExpertNum : moeExpertNum, H};
      std::vector<int64_t> expertScalesShape{BS, K};
      std::vector<int64_t> expandXShape{(TP_WORLD_SIZE > 0 ? TP_WORLD_SIZE : 1) * A, H};
      std::vector<int64_t> dynamicScalesShape{(TP_WORLD_SIZE > 0 ? TP_WORLD_SIZE : 1) * A};
      std::vector<int64_t> expandIdxShape{BS * K};
      std::vector<int64_t> expertTokenNumsShape{localExpertNum};
      std::vector<int64_t> epRecvCountsShape{(TP_WORLD_SIZE > 0 ? TP_WORLD_SIZE : 1) * localExpertNum * EP_WORLD_SIZE};
      std::vector<int64_t> tpRecvCountsShape{TP_WORLD_SIZE > 0 ? TP_WORLD_SIZE : 1};
      std::vector<int64_t> expandScalesShape{A};
  
      long long xShapeSize = GetShapeSize(xShape);
      long long expertIdsShapeSize = GetShapeSize(expertIdsShape);
      long long scalesShapeSize = GetShapeSize(scalesShape);
      long long expertScalesShapeSize = GetShapeSize(expertScalesShape);
      long long expandXShapeSize = GetShapeSize(expandXShape);
      long long dynamicScalesShapeSize = GetShapeSize(dynamicScalesShape);
      long long expandIdxShapeSize = GetShapeSize(expandIdxShape);
      long long expertTokenNumsShapeSize = GetShapeSize(expertTokenNumsShape);
      long long epRecvCountsShapeSize = GetShapeSize(epRecvCountsShape);
      long long tpRecvCountsShapeSize = GetShapeSize(tpRecvCountsShape);
      long long expandScalesShapeSize = GetShapeSize(expandScalesShape);
  
      // Construct host-side variables.
      std::vector<op::fp16_t> xHostData(xShapeSize, 1);
      std::vector<int32_t> expertIdsHostData;
      for (int32_t token_id = 0; token_id < expertIdsShape[0]; token_id++) {
          // Send each token to MoE experts {0, 1, ... k - 1}.
          for (int32_t k_id = 0; k_id < expertIdsShape[1]; k_id++) {
              expertIdsHostData.push_back(k_id);
          }
      }
      std::vector<float> scalesHostData(scalesShapeSize, 0);
      std::vector<float> expertScalesHostData(expertScalesShapeSize, 0);
      std::vector<op::fp16_t> expandXHostData(expandXShapeSize, 0);
      std::vector<float> dynamicScalesHostData(dynamicScalesShapeSize, 0);
      std::vector<int32_t> expandIdxHostData(expandIdxShapeSize, 0);
      std::vector<int64_t> expertTokenNumsHostData(expertTokenNumsShapeSize, 0);
      std::vector<int32_t> epRecvCountsHostData(epRecvCountsShapeSize, 0);
      std::vector<int32_t> tpRecvCountsHostData(tpRecvCountsShapeSize, 0);
      std::vector<float> expandScalesHostData(expandScalesShapeSize, 0);
  
      // Construct device-side variables.
      ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_BF16, &x);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expertIdsHostData, expertIdsShape, &expertIdsDeviceAddr, aclDataType::ACL_INT32, &expertIds);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(scalesHostData, scalesShape, &scalesDeviceAddr, aclDataType::ACL_FLOAT, &scales);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expertScalesHostData, expertScalesShape, &expertScalesDeviceAddr, aclDataType::ACL_FLOAT, &expertScales);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expandXHostData, expandXShape, &expandXDeviceAddr, (quantMode > 0) ? aclDataType::ACL_INT8 : aclDataType::ACL_BF16, &expandX);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(dynamicScalesHostData, dynamicScalesShape, &dynamicScalesDeviceAddr, aclDataType::ACL_FLOAT, &dynamicScales);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expandIdxHostData, expandIdxShape, &expandIdxDeviceAddr, aclDataType::ACL_INT32, &expandIdx);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expertTokenNumsHostData, expertTokenNumsShape, &expertTokenNumsDeviceAddr, aclDataType::ACL_INT64, &expertTokenNums);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(epRecvCountsHostData, epRecvCountsShape, &epRecvCountsDeviceAddr, aclDataType::ACL_INT32, &epRecvCounts);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(tpRecvCountsHostData, tpRecvCountsShape, &tpRecvCountsDeviceAddr, aclDataType::ACL_INT32, &tpRecvCounts);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
      ret = CreateAclTensor(expandScalesHostData, expandScalesShape, &expandScalesDeviceAddr, aclDataType::ACL_FLOAT, &expandScales);
      CHECK_RET(ret == ACL_SUCCESS, return ret);
  
      /* Declare the variables required for operator execution. */
      uint64_t dispatchWorkspaceSize = 0;
      aclOpExecutor *dispatchExecutor = nullptr;
      void *dispatchWorkspaceAddr = nullptr;
  
      uint64_t combineWorkspaceSize = 0;
      aclOpExecutor *combineExecutor = nullptr;
      void *combineWorkspaceAddr = nullptr;
  
      /* Execute the dispatch and combine operators in sequence. */
      // Call the first-phase API of the dispatch operator.
      ret = aclnnMoeDistributeDispatchGetWorkspaceSize(
          x, expertIds,
          (quantMode > 0 ? scales : nullptr), nullptr,
          expertScales,
          hcomEpName, EP_WORLD_SIZE, args.epRankId,
          moeExpertNum, hcomTpName, TP_WORLD_SIZE,
          args.tpRankId, expertShardType, sharedExpertNum,
          sharedExpertRankNum, quantMode, globalBS,
          expertTokenNumsType,
          expandX, dynamicScales,
          expandIdx, expertTokenNums,
          epRecvCounts, tpRecvCounts,
          expandScales, &dispatchWorkspaceSize,
          &dispatchExecutor
      );
      CHECK_RET(
          ret == ACL_SUCCESS,
          LOG_PRINT("[ERROR] aclnnMoeDistributeDispatchGetWorkspaceSize failed. ret = %d\n", ret); return ret
      );
      // Apply for device memory based on the workspaceSize calculated by the first-phase API of the dispatch operator.
      if (dispatchWorkspaceSize > 0) {
          ret = aclrtMalloc(&dispatchWorkspaceAddr, dispatchWorkspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtMalloc failed. ret = %d\n", ret); return ret);
      }
      // Call the second-phase API of the dispatch operator.
      ret = aclnnMoeDistributeDispatch(dispatchWorkspaceAddr, dispatchWorkspaceSize, dispatchExecutor, args.dispatchStream);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclnnMoeDistributeDispatch failed. ret = %d\n", ret); return ret);
      // (Boilerplate) Synchronously wait for task execution to complete.
      ret = aclrtSynchronizeStreamWithTimeout(args.dispatchStream, 10000);
      CHECK_RET(
          ret == ACL_SUCCESS,
          LOG_PRINT("[ERROR] aclrtSynchronizeStreamWithTimeout failed. ret = %d\n", ret); return ret
      );
  
      // Call the first-phase API of the combine operator.
      ret = aclnnMoeDistributeCombineGetWorkspaceSize(expandX, expertIds, expandIdx, epRecvCounts, expertScales, tpRecvCounts,
          nullptr, nullptr, nullptr, nullptr, nullptr,
          hcomEpName, EP_WORLD_SIZE, args.epRankId, moeExpertNum, hcomTpName, TP_WORLD_SIZE, args.tpRankId,
          expertShardType, sharedExpertNum, sharedExpertRankNum, globalBS, outDtype, commQuantMode, groupListType,
          x, &combineWorkspaceSize, &combineExecutor);
      CHECK_RET(
          ret == ACL_SUCCESS,
          LOG_PRINT("[ERROR] aclnnMoeDistributeCombineGetWorkspaceSize failed. ret = %d\n", ret); return ret
      );
      // Apply for device memory based on the workspaceSize calculated by the first-phase API of the combine operator.
      if (combineWorkspaceSize > 0) {
          ret = aclrtMalloc(&combineWorkspaceAddr, combineWorkspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtMalloc failed. ret = %d\n", ret); return ret);
      }
      // Call the second-phase API of the combine operator.
      ret = aclnnMoeDistributeCombine(combineWorkspaceAddr, combineWorkspaceSize, combineExecutor, args.combineStream);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclnnMoeDistributeCombine failed. ret = %d\n", ret); return ret);
      // (Boilerplate) Synchronously wait for task execution to complete.
      ret = aclrtSynchronizeStreamWithTimeout(args.combineStream, 10000);
      CHECK_RET(
          ret == ACL_SUCCESS,
          LOG_PRINT("[ERROR] aclrtSynchronizeStreamWithTimeout failed. ret = %d\n", ret); return ret
      );
  
      LOG_PRINT("[INFO] device_%d aclnnMoeDistributeDispatch and aclnnMoeDistributeCombine execute successfully.\n", args.rankId);
  
      // Release device resources.
      if (dispatchWorkspaceSize > 0) {
          aclrtFree(dispatchWorkspaceAddr);
      }
      if (combineWorkspaceSize > 0) {
          aclrtFree(combineWorkspaceAddr);
      }
      if (x != nullptr) {
          aclDestroyTensor(x);
      }
      if (expertIds != nullptr) {
          aclDestroyTensor(expertIds);
      }
      if (scales != nullptr) {
          aclDestroyTensor(scales);
      }
      if (expertScales != nullptr) {
          aclDestroyTensor(expertScales);
      }
      if (expandX != nullptr) {
          aclDestroyTensor(expandX);
      }
      if (dynamicScales != nullptr) {
          aclDestroyTensor(dynamicScales);
      }
      if (expandIdx != nullptr) {
          aclDestroyTensor(expandIdx);
      }
      if (expertTokenNums != nullptr) {
          aclDestroyTensor(expertTokenNums);
      }
      if (epRecvCounts != nullptr) {
          aclDestroyTensor(epRecvCounts);
      }
      if (tpRecvCounts != nullptr) {
          aclDestroyTensor(tpRecvCounts);
      }
      if (expandScales != nullptr) {
          aclDestroyTensor(expandScales);
      }
      if (xDeviceAddr != nullptr) {
          aclrtFree(xDeviceAddr);
      }
      if (expertIdsDeviceAddr != nullptr) {
          aclrtFree(expertIdsDeviceAddr);
      }
      if (scalesDeviceAddr != nullptr) {
          aclrtFree(scalesDeviceAddr);
      }
      if (expertScalesDeviceAddr != nullptr) {
          aclrtFree(expertScalesDeviceAddr);
      }
      if (expandXDeviceAddr != nullptr) {
          aclrtFree(expandXDeviceAddr);
      }
      if (dynamicScalesDeviceAddr != nullptr) {
          aclrtFree(dynamicScalesDeviceAddr);
      }
      if (expandIdxDeviceAddr != nullptr) {
          aclrtFree(expandIdxDeviceAddr);
      }
      if (expertTokenNumsDeviceAddr != nullptr) {
          aclrtFree(expertTokenNumsDeviceAddr);
      }
      if (epRecvCountsDeviceAddr != nullptr) {
          aclrtFree(epRecvCountsDeviceAddr);
      }
      if (expandScalesDeviceAddr != nullptr) {
          aclrtFree(expandScalesDeviceAddr);
      }
      if (tpRecvCountsDeviceAddr != nullptr) {
          aclrtFree(tpRecvCountsDeviceAddr);
      }
  
      HcclCommDestroy(args.hcclEpComm);
      HcclCommDestroy(args.hcclTpComm);
      aclrtDestroyStream(args.dispatchStream);
      aclrtDestroyStream(args.combineStream);
      aclrtDestroyContext(args.context);
      aclrtResetDevice(args.rankId);
  
      return 0;
  }
  
  int run_example_on_A2(int rankId, const char* RANK_TABLE_FILE, const char* FIRST_RANK_ID)
  {
      Args args;
      aclrtStream dispatchStream;
      aclrtStream combineStream;
      aclrtContext context;
  
      int ret = aclrtSetDevice(rankId);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtSetDevice failed. ret = %d", ret));
      ret = aclrtCreateContext(&context, rankId);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateContext failed. ret = %d", ret));
      ret = aclrtCreateStream(&dispatchStream);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateStream failed. ret = %d", ret));
      ret = aclrtCreateStream(&combineStream);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateStream failed. ret = %d", ret));
  
      int first_rank_id = std::stoi(std::string(FIRST_RANK_ID));
      HcclComm hcclComm = nullptr;
      int rank_id = rankId + first_rank_id;
      ret = HcclCommInitClusterInfo(RANK_TABLE_FILE, rank_id, &hcclComm);
      if (ret != HCCL_SUCCESS) {
          std::cout << "[ERROR] HCCL CommInitClusterInfo failed. ret = " << ret << std::endl;
          return ret;
      }
      std::cout << "[INFO] HcclCommInitClusterInfo success, rank_id:" << rank_id << ", rankSize:" << DEV_NUM
              << ", hcclComm:" << hcclComm << std::endl;
      uint32_t epRankId = rank_id / TP_WORLD_SIZE;
      uint32_t tpRankId = rank_id % TP_WORLD_SIZE;
  
      args.rankId = rankId;
      args.epRankId = epRankId;
      args.tpRankId = tpRankId;
      args.hcclEpComm = hcclComm;
      args.dispatchStream = dispatchStream;
      args.combineStream = combineStream;
      args.context = context;
  
      launchOneThreadDispatchAndCombine(args);
      return 0;
  }
  
  int run_example_on_A3A5()
  {
      int ret = aclInit(nullptr);
      CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclInit failed. ret = %d\n", ret); return ret);
  
      aclrtStream dispatchStream[DEV_NUM];
      aclrtStream combineStream[DEV_NUM];
      aclrtContext context[DEV_NUM];
      for (uint32_t rankId = 0; rankId < DEV_NUM; rankId++) {
          ret = aclrtSetDevice(rankId);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtSetDevice failed. ret = %d\n", ret); return ret);
          ret = aclrtCreateContext(&context[rankId], rankId);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateContext failed. ret = %d\n", ret); return ret);
          ret = aclrtCreateStream(&dispatchStream[rankId]);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateStream failed. ret = %d\n", ret); return ret);
          ret = aclrtCreateStream(&combineStream[rankId]);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclrtCreateStream failed. ret = %d\n", ret); return ret);
      }
  
      int32_t devicesEp[TP_WORLD_SIZE][EP_WORLD_SIZE];
      for (int32_t tpId = 0; tpId < TP_WORLD_SIZE; tpId++) {
          for (int32_t epId = 0; epId < EP_WORLD_SIZE; epId++) {
              devicesEp[tpId][epId] = epId * TP_WORLD_SIZE + tpId;
          }
      }
      // Initialize the EP communication domain, ep = 8 {0,2,4,6,8,10,12,14} {1,3,5,7,9,11,13,15}.
      HcclComm commsEp[TP_WORLD_SIZE][EP_WORLD_SIZE];
      for (int32_t tpId = 0; tpId < TP_WORLD_SIZE; tpId++) {
          ret = HcclCommInitAll(EP_WORLD_SIZE, devicesEp[tpId], commsEp[tpId]);
          CHECK_RET(
              ret == ACL_SUCCESS,
              LOG_PRINT("[ERROR] HcclCommInitAll ep world %d failed. ret = %d\n", tpId, ret); return ret
          );
      }
  
      int32_t devicesTp[EP_WORLD_SIZE][TP_WORLD_SIZE];
      for (int32_t epId = 0; epId < EP_WORLD_SIZE; epId++) {
          for (int32_t tpId = 0; tpId < TP_WORLD_SIZE; tpId++) {
              devicesTp[epId][tpId] = epId * TP_WORLD_SIZE + tpId;
          }
      }
      // Initialize the TP communication domain, tp = 2 {0,1} {2,3} {4,5} {6,7} {8,9} {10,11} {12,13} {14,15}.
      HcclComm commsTp[EP_WORLD_SIZE][TP_WORLD_SIZE];
      for (int32_t epId = 0; epId < EP_WORLD_SIZE; epId++) {
          ret = HcclCommInitAll(TP_WORLD_SIZE, devicesTp[epId], commsTp[epId]);
          CHECK_RET(
              ret == ACL_SUCCESS,
              LOG_PRINT("[ERROR] HcclCommInitAll tp world %d failed. ret = %d\n", epId, ret); return ret
          );
      }
  
      Args args[DEV_NUM];
      // Each thread calls the corresponding device to execute the operator.
      std::vector<std::unique_ptr<std::thread>> threads(DEV_NUM);
      for (uint32_t rankId = 0; rankId < DEV_NUM; rankId++) {
          uint32_t epRankId = rankId / TP_WORLD_SIZE;
          uint32_t tpRankId = rankId % TP_WORLD_SIZE;
  
          args[rankId].rankId = rankId;
          args[rankId].epRankId = epRankId;
          args[rankId].tpRankId = tpRankId;
          args[rankId].hcclEpComm = commsEp[tpRankId][epRankId];
          args[rankId].hcclTpComm = commsTp[epRankId][tpRankId];
          args[rankId].dispatchStream = dispatchStream[rankId];
          args[rankId].combineStream = combineStream[rankId];
          args[rankId].context = context[rankId];
          threads[rankId].reset(new(std::nothrow) std::thread(&launchOneThreadDispatchAndCombine, std::ref(args[rankId])));
      }
      for (uint32_t rankId = 0; rankId < DEV_NUM; rankId++) {
          threads[rankId]->join();
      }
      aclFinalize();
      LOG_PRINT("[INFO] aclFinalize success\n");
      return 0;
  }
  
  int main(int argc, char *argv[])
  {
      const char* env_var_name = "RANK_TABLE_FILE and FIRST_RANK_ID";
      if (!env_dev_num) {
          LOG_PRINT("[WARNING] Please check whether environment variable ENV_DEV_NUM is set correctly.\n");
          return 0;
      }
      int actual_env_dev_num = std::stoi(std::string(env_dev_num));
      if (actual_env_dev_num < DEV_NUM) {
          LOG_PRINT("[INFO] ENV_DEV_NUM = %d is less than %d, currently not supported\n", actual_env_dev_num, DEV_NUM);
          return 0;
      }
      if (!rank_table_file && !first_rank_id) {
          EP_WORLD_SIZE = 8;
          TP_WORLD_SIZE = 2;
          DEV_NUM = EP_WORLD_SIZE * TP_WORLD_SIZE;
          LOG_PRINT("[INFO] %s are not identified and example on <Atlas A3> will be executed!\n", env_var_name);
          int ret = run_example_on_A3A5();
      }
      else if (rank_table_file && !first_rank_id) {
          EP_WORLD_SIZE = 2;
          TP_WORLD_SIZE = 1;
          DEV_NUM = 2;
          LOG_PRINT("[INFO] %s are not identified and example on <Atlas A5> will be executed!\n", env_var_name);
          int ret = run_example_on_A3A5();
      }
      else if (rank_table_file && first_rank_id) {
          EP_WORLD_SIZE = 16;
          TP_WORLD_SIZE = 1;
          DEV_NUM = EP_WORLD_SIZE;
          LOG_PRINT("[INFO] %s are identified and example on <Atlas A2> will be executed!\n", env_var_name);
          uint32_t single_machine_dev_num = EP_WORLD_SIZE / MACHINE_NUM;
          std::vector<std::unique_ptr<std::thread>> threads(single_machine_dev_num);
          auto ret = aclInit(nullptr);
          CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] aclInit failed. ret = %d\n", ret); return ret);
          for (int rankId = 0; rankId < single_machine_dev_num; ++rankId) {
              threads[rankId] = std::make_unique<std::thread>([rankId,&ret]()
              {
                  ret = run_example_on_A2(rankId, rank_table_file, first_rank_id);
                  CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("[ERROR] run example on A2 failed. ret = %d\n", ret); return ret);
              });
          }
          for (int rankId = 0; rankId < single_machine_dev_num; ++rankId) {
              threads[rankId]->join();
          }
          aclFinalize();
          LOG_PRINT("[INFO] aclFinalize success\n");
      } else {
          LOG_PRINT("[WARNING] Please check whether %s are set correctly.\n", env_var_name);
      }
  
      return 0;
  }