Documentation
Download

AllReduceOperation

Description

Computes the data on multiple communication cards, including addition, multiplication, minimum, and maximum calculations, and sends the result to each card.

This operator involves multi-device operations. You can configure HCCL environment variables as required. For details, see . For details about how to configure TLS, see the HCCN Tool API Reference of the corresponding device.

Application Scenarios

Many-to-many: The same Reduce operation (sum, min, max, or prod) is performed on all nodes, and the result obtained by the data reduction operation of all nodes is sent to all nodes (broadcast).

Figure 1 Working principle of the AllReduce operator

Application scenarios:

  • Reduce operation in the combination of broadcast and reduce in AllReduce.
  • Reduce operation in the ReduceScatter combination.
  • The master node in the distributed training parameter server structure broadcasts data to the worker node, and then reduces data from the worker node to the master node.

Example:

  • Non-quantization scenario (rankSize=2):
    • Scenario 1: allReduce sum

      npu0:

      x=[-4,-50]
output=[-8,-100]

      npu1:

      x=[-4,-50]
output=[-8,-100]
    • Scenario 2: allReduce max

      npu0:

      x=[4,20]
output=[10,50]

      npu1:

      x=[10,50]
output=[10,50]
    • Scenario 3: allReduce min

      npu0:

      x=[3,100]
output=[3,20]

      npu1:

      x=[3,20]
output=[3,20]
    • Scenario 4: allReduce prod

      npu0:

      x=[-93,-48]
output=[8649,2304]

      npu1:

      x=[-93,-48]
output=[8649,2304]
  • Quantization scenario: Quantization transmission (rankSize=2) is performed during allReduce sum.
  • Quantization calculation formula:

  • Scenario 1: QUANT_TYPE_PER_TENSOR quantizes the entire tensor.
    x=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
scale=[1]
offset=[0]
output=[2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32]
  • Scenario 2: QUANT_TYPE_PER_CHANNEL quantizes each channel in the tensor.
    x=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
scale=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]
offset=[0]
output=[2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32]

Definition

struct AllReduceParam {
    enum QuantType : int {
        QUANT_TYPE_UNDEFINED = 0,
        QUANT_TYPE_PER_TENSOR = 1,
        QUANT_TYPE_PER_CHANNEL = 2,
        QUANT_TYPE_MAX = 3,  
    };
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string allReduceType = "sum";
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    QuantType quantType = QUANT_TYPE_UNDEFINED;
    aclDataType outDataType = ACL_DT_UNDEFINED;
    uint8_t rsv[64] = {0};
};

Parameters

Member

Type

Default Value

Description

rank

int

0

Communication ID of the current card

rankSize

int

0

Number of communication cards

rankRoot

int

0

Primary communication rank.

allReduceType

string

"sum"

Computation type.

The value can be sum, prod, max, or min.

backend

string

"hccl"

Communication computation type. Only hccl and lccl are supported.

For the Atlas inference products, backend can only be hccl.

  • When backend is set to hccl:
    • The operation can be sum, prod, max, or min.
    • If allReduceType is set to prod, the data type cannot be int16 or bf16.
    • The Atlas inference products does not support int64 and bf16. The int16 data type is supported only when allReduceType is set to sum.
  • When backend is set to lccl:
    • The operation can be sum, max, or min.
    • The int64 data type is not supported.
    • If the machine topology is the single-server 16-card topology with the Atlas A2 inference products, only full topology communication of 16 cards or communication of any card on a single node is supported.

hcclComm

HcclComm

nullptr

Pointer to the HCCL communicator.

By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.

commMode

CommMode

COMM_MULTI_PROCESS

Communication mode. For details, see CommMode. In the scenario where hccl multi-thread is applied, a communicator can only be imported externally.

rankTableFile

string

-

Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported.

If a ranktable is configured for a single-node system, the ranktable is used to initialize the communicator.

For details, see .

commDomain

string

-

Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain must be set to a number ranging from 0 to 63. When commMode is multi-thread, deterministic computing is not supported. In this case, LCCL_DETERMINISTIC must be set to 0 or false.

In the multi-process/multi-thread multi-communicator concurrency scenario, LCCL_PARALLEL must be set to 1 or true. After the multi-communicator parallelism is used, LCCL_PARALLEL must be set to 0 or false. Otherwise, the performance in basic scenarios deteriorates.

quantType

QuantType

QUANT_TYPE_UNDEFINED

Quantization type.

QUANT_TYPE_UNDEFINED: default value

QUANT_TYPE_PER_TENSOR: quantization of the entire tensor

QUANT_TYPE_PER_CHANNEL: quantization of each channel in the tensor.

QUANT_TYPE_MAX: maximum value of enumerated types

outDataType

aclDataType

ACL_DT_UNDEFINED
  • For floating-point AllReduce, if outDataType is set to ACL_DT_UNDEFINED, the data type of the output tensor is the same as that of the input tensor.
  • For AllReduce quantization, if the data type of the output tensor is different from that of the input tensor, set outDataType to the expected data type of the output tensor. Only ACL_FLOAT16 can be configured for quantization.

rsv[64]

uint8_t

{0}

Reserved

Note: When multiple LCCL communicators are running concurrently, the maximum communication data volume cannot exceed 100 MB. If the data volume is greater than 1 MB, the performance deteriorates. During concurrent communication, bandwidth sharing is used, and the performance deterioration is incompatible with the deterministic computing function. After deterministic computing is enabled, this function becomes invalid.

Function Input and Output Description

Parameter

Dimension

Data Type

Format

Description

x

[-1,…,-1]

The value -1 indicates that the size of the current dimension is not restricted.
  • "hccl": float16/float/int8/int16/int32/int64/bf16
  • "lccl": float16/float/int8/int16/int32/bf16

ND

Input tensor

output

[-1,…,-1]

The value -1 indicates that the size of the current dimension is not restricted.
  • "hccl": float16/float/int8/int16/int32/int64/bf16
  • "lccl": float16/float/int8/int16/int32/bf16

ND

Output tensor.

Input and Output (quantType = QUANT_PER_Tensor)

Parameter

Dimension

Data Type

Format

Description

x

[-1,...,-1,n]

The value -1 indicates that the size of the current dimension is not restricted.

int8

ND

Input vector. The value of the last dimension n is an integer multiple of 16.

scale

[1]

float16

ND

Input. The elements in scale must be scalars.

offset

[1]

float16

ND

Input. The elements in offset must be scalars.

y

[-1,...,-1,n]

The value -1 indicates that the size of the current dimension is not restricted.

float16

ND

Quantization output result. The shape of x is the same as that of y.

Input and Output (quantType = QUANT_PER_Cannel)

Parameter

Dimension

Data Type

Format

Description

x

[-1,...,-1,n]

The value -1 indicates that the size of the current dimension is not restricted.

int8

ND

Input vector. The value of the last dimension n is an integer multiple of 16.

scale

[1, n] or [n]

float16

ND

Input. The element in scale cannot be 0. The size of the last dimension n is an integer multiple of 16.

offset

[1]

float16

ND

Input. The elements in offset must be scalars.

y

[-1,...,-1,n]

The value -1 indicates that the size of the current dimension is not restricted.

float16

ND

Quantization output result. The shape of x is the same as that of y.

Restrictions

rank, rankSize, and rankRoot must meet the following conditions:

  • 0 ≤ rank < rankSize
  • 0 ≤ rankRoot < rankSize
  • If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
  • When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
    rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
  • Currently, multiple HCCP process instances cannot run on a single device, and multiple communication operators cannot run on a single device at the same time. For example, when a communication operator is running, multiple models cannot run on the same card.
Parent topic: atb/infer_op_params.h
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details