HCCL_ALGO

Description

Configures the communication algorithms between servers and supernodes. The algorithms can be configured globally or by operator.

This section only provides a brief introduction to configurable algorithm functions. For details, see Collective Communication Algorithms.

• HCCL provides the adaptive algorithm selection function, so that it can select an appropriate algorithm by default based on the product form, data size, and number of servers, requiring no manual configuration. Enabling this environment variable to specify the inter-server or inter-supernode communication algorithm would invalidate the adaptive algorithm selection function.
• In some communication operators, when a specific type of AI processor is used and the data size is small, the communication algorithm is adaptively selected by HCCL and not controlled by this environment variable.
• The algorithms listed in this section cover the full set of communication algorithms that can be configured in HCCL. For details about the inter-server communication algorithms and inter-supernode communication algorithms supported by specific products, see List of Supported Algorithms for Inter-Server Communication and List of Supported Algorithms for Inter-Supernode Communication.

• The global configuration method is as follows:

  export HCCL_ALGO="level0:NA;level1:<algo>;level2:<algo>"

  • level0 indicates the intra-server communication algorithm. Currently, this value can only be set to NA.
  • level1 indicates the inter-server communication algorithm. This value can be set to the following:
    • ring: a communication algorithm based on the ring topology. It features a large number of communication steps (linear complexity) and relatively high latency. However, it has a simple communication relationship and is less affected by network congestion. This algorithm applies to scenarios featuring a small number of servers within the communicator, low communication data volume, severe network congestion, and inapplicability of the pipeline algorithm.
    • H-D_R: Recursive Halving-Doubling (RHD) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. However, extra traffic is introduced if the number of nodes is not an integral power of 2. This algorithm applies to scenarios where the number of servers in the communicator is an integral power of 2 with the pipeline algorithm inapplicable, or where the number of servers is not an integral power of 2 but the communication data volume is small.
    • NHR: Nonuniform Hierarchical Ring (NHR) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. This algorithm applies to scenarios where there are lots of servers in the communicator and the pipeline algorithm is not applicable.

      For the Atlas 350 Accelerator Card of the current version, only the NHR algorithm can be configured.

    • NHR_V1: NHR algorithm of an earlier version. It features a small number of communication steps (root complexity) and relatively low latency. This algorithm applies to scenarios where the number of servers in the communicator is not an integer power of 2 and the pipeline algorithm is not applicable. Theoretically, the performance of the NHR_V1 algorithm is lower than that of the new NHR algorithm. This configuration option will be deprecated in the future. It is recommended that developers use the NHR algorithm.
    • NB: Nonuniform Bruck (NB) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. This algorithm applies to scenarios where there are lots of servers in the communicator and the pipeline algorithm is not applicable.
    • AHC: Asymmetric Hierarchical Concatenate (AHC) algorithm. This algorithm applies to scenarios where NPUs are symmetrically or asymmetrically distributed (that is, asymmetric number of devices) across multiple layers in the communicator. When bandwidth convergence exists between layers in the communicator, the relative benefits are better.

      Note: When level1 (inter-server communication algorithm) is set to AHC, level2 (inter-supernode communication algorithm) automatically uses the AHC algorithm without additional configuration. Even if level2 is set to another algorithm, the setting does not take effect.

    • pipeline: pipeline parallel algorithm. It can concurrently use the intra-server and inter-server links. This algorithm applies to scenarios where the communication data size is large and each server in the communicator contains multiple devices.
    • pairwise: pairwise communication algorithm. It is used only for the AlltoAll, AlltoAllV, and AlltoAllVC operators. It features a large number of communication steps (linear complexity) and relatively high latency, and requires additional memory allocation. The memory size is in direct proportion to the data size. However, it can avoid the problem of one-to-many network. This algorithm applies to scenarios where the communication data size is large and the problem of one-to-many network needs to be avoided.
    If level1 is not set:

    • For the Atlas 350 Accelerator Card, the NHR algorithm is used by default.
    • For the Atlas A3 training product/Atlas A3 inference product, the algorithm is automatically selected based on the product form, number of nodes, and data size.
    • For the Atlas A2 training product/Atlas A2 inference product, the algorithm is automatically selected based on the product form, number of nodes, and data size.
    • For the Atlas training products, if the number of servers in a communicator is not an integral power of 2, the ring algorithm is used by default. In other scenarios, the H-D_R algorithm is used by default.
  • level2 indicates the inter-supernode communication algorithm. This value can be set to the following:
    • ring: a communication algorithm based on the ring topology. It features a large number of communication steps (linear complexity) and relatively high latency. However, it has a simple communication relationship and is less affected by network congestion. This algorithm applies to scenarios where the number of supernodes in the communicator is small and is not an integral power of 2.
    • H-D_R: Recursive Halving-Doubling (RHD) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. However, extra traffic is introduced if the number of nodes is not an integral power of 2. This algorithm applies to scenarios where the number of supernodes in the communicator is an integral power of 2, or the number of servers is not an integral power of 2 but the communication data size is small.
    • NHR: Nonuniform Hierarchical Ring (NHR) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. This algorithm applies to scenarios where the number of supernodes in the communicator is large.
    • NB: Nonuniform Bruck (NB) algorithm. It features a small number of communication steps (logarithmic complexity) and relatively low latency. This algorithm applies to scenarios where the number of supernodes in the communicator is large.
    • pipeline: pipeline parallel algorithm. It can concurrently use the intra-supernode and inter-supernode links. This algorithm applies to scenarios where the communication data size is large and each supernode in the communicator contains multiple devices.

    For details about the communication operators, data types, and network running modes supported by each inter-supernode communication algorithm, see List of Supported Algorithms for Inter-Supernode Communication.

    If level2 is not set, the ring algorithm is used when the number of supernodes in the communicator is less than 8 and is not an integral power of 2. In other scenarios, the H-D_R algorithm is used.

    Currently, the level2 configuration applies only to the following scenarios:

    • It is supported only by the Atlas A3 training product/Atlas A3 inference product.
    • Only scenarios where the communication operator expansion mode is set to AI_CPU are supported. The communication operator expansion mode can be configured via the environment variable HCCL_OP_EXPANSION_MODE.
• Configure the communication algorithm based on the operator type as follows:

  export HCCL_ALGO="<op0>=level0:NA;level1:<algo0>;level2:<algo1>/<op1>=level0:NA;level1:<algo3>;level2:<algo4>"

  Where:

  • <op> indicates the type of the communication operator. The options are as follows:
    • allgather: corresponds to the communication operators AllGather and AllGatherV.
    • reducescatter: corresponds to the communication operators ReduceScatter and ReduceScatterV.
    • allreduce: corresponds to the communication operator AllReduce.
    • broadcast: corresponds to the communication operator Broadcast.
    • reduce: corresponds to the communication operator Reduce.
    • scatter: corresponds to the communication operator Scatter.
    • alltoall: corresponds to the communication operators AlltoAll, AlltoAllV, and AlltoAllVC.
  • <algo> specifies the communication algorithm used by the specified communication operator. The supported configuration is the same as that of level1 and level2 in the global configuration method. Ensure that the specified communication algorithm is supported by the target communication operator. For details about the communication operators supported by each algorithm, see List of Supported Algorithms for Inter-Server Communication and List of Supported Algorithms for Inter-Supernode Communication. If no communication algorithm is specified for a communication operator, the system automatically selects an appropriate algorithm according to the product form, node count, and communication data volume.
  • Use slashes (/) to separate the configurations of multiple operators.

Example

• Global configuration

  export HCCL_ALGO="level0:NA;level1:NHR"

• Configuration by operator

  # The AllReduce operator uses the ring algorithm and the AllGather operator uses the RHD algorithm. Other operators automatically select a communication algorithm based on the product form, number of nodes, and data size.
  export HCCL_ALGO="allreduce=level0:NA;level1:ring/allgather=level0:NA;level1:H-D_R"

Constraints

• In the current version, the intra-server communication algorithm can only be set to NA.
• For the Atlas A2 training product/Atlas A2 inference product, you are advised not to configure the HCCL_ALGO environment variable in the order-preserving scenario of strict deterministic computation.
• If you call the HCCL C APIs to initialize a communicator with specific configurations and specify the communication algorithm using the hcclAlgo parameter in HcclCommConfig, the configuration at the communicator granularity takes precedence.

Applicability

Atlas 350 Accelerator Card

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product

Atlas training product