AllReduceOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	√
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	√
Atlas inference products	√
Atlas 200I/500 A2 inference products	x

Description

Computes the data on multiple communication cards, including addition, multiplication, minimum, and maximum calculations, and sends the result to each card.

This operator involves multi-device operations. You can configure HCCL environment variables as required. For details, see Collective Communication in CANN Environment Variables.

Application Scenarios

Many-to-many: The same Reduce operation (sum, min, max, or prod) is performed on all nodes, and the result obtained by the data reduction operation of all nodes is sent to all nodes (broadcast).

Figure 1 Working principle of the AllReduce operator

Application scenarios:

Reduce operation in the combination of broadcast and reduce in AllReduce.
Reduce operation in the ReduceScatter combination.
The master node in the distributed training parameter server structure broadcasts data to the worker node, and then reduces data from the worker node to the master node.

Example:

Non-quantization scenario (rankSize=2):

Scenario 1: allReduce sum

npu0:

x=[-4,-50]
output=[-8,-100]

npu1:

x=[-4,-50]
output=[-8,-100]

Scenario 2: allReduce max

npu0:

x=[4,20]
output=[10,50]

npu1:

x=[10,50]
output=[10,50]

Scenario 3: allReduce min

npu0:

x=[3,100]
output=[3,20]

npu1:

x=[3,20]
output=[3,20]

Scenario 4: allReduce prod

npu0:

x=[-93,-48]
output=[8649,2304]

npu1:

x=[-93,-48]
output=[8649,2304]

Quantization scenario: Quantization transmission (rankSize=2) is performed during allReduce sum.
- Quantization calculation formula:
  $\text{[math]}$
- Scenario 1: QUANT_TYPE_PER_TENSOR quantizes the entire tensor.
```
x=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
scale=[1]
offset=[0]
output=[2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32]
```
- Scenario 2: QUANT_TYPE_PER_CHANNEL quantizes each channel in the tensor.
```
x=[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
scale=[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1]
offset=[0]
output=[2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32]
```

Definition

struct AllReduceParam {
    enum QuantType : int {
        QUANT_TYPE_UNQUANT = 0,
        QUANT_TYPE_UNDEFINED = 0,
        QUANT_TYPE_PER_TENSOR = 1,
        QUANT_TYPE_PER_CHANNEL = 2,
        QUANT_TYPE_MAX = 3,  
    };
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string allReduceType = "sum";
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    QuantType quantType = QUANT_TYPE_UNDEFINED;
    aclDataType outDataType = ACL_DT_UNDEFINED;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Value Range	Mandatory	Description
rank	int	0	[0, rankSize-1]	Yes	Rank of the current device within the communicator.
rankSize	int	0	-	Yes	Number of communication cards
rankRoot	int	0	[0, rankSize-1]	Yes	Root rank for the operation.
allReduceType	string	sum	sum prod max min	Yes	Computation type. The value can be sum, prod, max, or min. allReduceType indicates the reduce operation type (OperationType) of collective communication. For details, see HcclReduceOp in Huawei Collective Communication Library (HCCL). prod (multiplication) is not supported for Atlas 350 accelerator cards.
backend	string	hccl	lccl/hccl	Yes	Communication computation type. Only hccl and lccl are supported. The LCCL function will be deleted in later versions. You are advised to use the HCCL function. For Atlas inference products and Atlas 350 accelerator cards, backend can only be hccl. When backend is set to hccl: sum, prod, max, and min are supported. On Atlas A2 training products/Atlas A2 inference products, if allReduceType is set to prod, the data type cannot be int16 or bf16. Atlas inference products do not support int64 and bf16. The int16 data type is supported only when allReduceType is set to sum. Atlas training products do not support int16 and bf16. Quantization is not supported. When backend is set to lccl: sum, max, and min are supported. The int64 data type is not supported. For Atlas 200T A2 Box16 heterogeneous subrack, the communicator can be created only among the first eight or last eight devices. A communicator that contains both the first eight and last eight devices is not supported. Deterministic computation is involved. You can enable or disable it by using the environment variable LCCL_DETERMINISTIC.
hcclComm	HcclComm	nullptr	-	No	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	COMM_MULTI_PROCESS COMM_MULTI_THREAD	No	Communication mode. For details, see CommMode. In the scenario where hccl multi-thread is applied, a communicator can only be imported externally.
rankTableFile	string	-	-	No	Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported. If a rank table is configured for a single-node system, the rank table is used to initialize the communicator. For details, see 《HCCL集合通信库》中的“ranktable文件配置资源信息”章节.
commDomain	string	-	-	No	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain must be set to a number ranging from 0 to 65535. When commMode is multi-thread, deterministic computing is not supported. In this case, LCCL_DETERMINISTIC must be set to 0 or false. In the multi-process/multi-thread multi-communicator concurrency scenario, LCCL_PARALLEL must be set to 1 or true. After the multi-communicator parallelism is used, LCCL_PARALLEL must be set to 0 or false. Otherwise, the performance in basic scenarios deteriorates. The memory size of the communicator can be set for performance optimization. The format is commDomain:commDomain size (MB). For example, 0:400. When the configurations of multiple operators in the same communicator conflict, the configuration of the first operator executed in the communicator is used. Restriction: The default value is 200. Currently, the value cannot be less than 200.
quantType	QuantType	QUANT_TYPE_UNQUANT	QUANT_TYPE_UNQUANT QUANT_TYPE_UNDEFINED QUANT_TYPE_PER_TENSOR QUANT_TYPE_PER_CHANNEL QUANT_TYPE_MAX	No	Quantization type. QUANT_TYPE_UNQUANT: default value. QUANT_TYPE_UNDEFINED: default value QUANT_TYPE_PER_TENSOR: quantization of the entire tensor QUANT_TYPE_PER_CHANNEL: quantization of each channel in the tensor. QUANT_TYPE_MAX: maximum value of enumerated types
outDataType	aclDataType	ACL_DT_UNDEFINED	ACL_DT_UNDEFINED ACL_FLOAT16	No	For floating-point AllReduce, if outDataType is set to ACL_DT_UNDEFINED, the data type of the output tensor is the same as that of the input tensor. For AllReduce quantization, if the data type of the output tensor is different from that of the input tensor, set outDataType to the expected data type of the output tensor. Only ACL_FLOAT16 can be configured for quantization.
rsv[64]	uint8_t	{0}	[0]	No	Reserved

Note: When multiple LCCL communicators are running concurrently, the maximum communication data volume cannot exceed 100 MB. If the data volume is greater than 1 MB, the performance deteriorates. During concurrent communication, the bandwidth is shared and the performance deteriorates. LCCL multi-communicator parallelism is incompatible with deterministic computing. After deterministic computing is enabled, this function becomes invalid.

Function Input and Output Description

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	hccl: float16/float/int8/int16/int32/int64/bf16 lccl: float16/float/int8/int16/int32/bf16	ND	Input tensor
output	[dim_0, dim_1, ..., dim_n]	hccl: float16/float/int8/int16/int32/int64/bf16 lccl: float16/float/int8/int16/int32/bf16	ND	Output tensor.

Input and Output (quantType = QUANT_PER_Tensor)

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	int8	ND	Input tensor. The value of the last dimension n is an integer multiple of 16.
scale	[1]	float16	ND	Input. The elements in scale must be scalars.
offset	[1]	float16	ND	Input. The elements in offset must be scalars.
y	[dim_0, dim_1, ..., dim_n]	float16	ND	Quantization output result. The shape of x is the same as that of y.

Input and Output (quantType = QUANT_PER_Channel)

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	int8	ND	Input tensor. The value of the last dimension n is an integer multiple of 16. n cannot be greater than 4194304.
scale	[1, n] or [n]	float16	ND	Input. The element in scale cannot be 0. The size of the last dimension n is an integer multiple of 16.
offset	[1]	float16	ND	Input. The elements in offset must be scalars.
y	[dim_0, dim_1, ..., dim_n]	float16	ND	Quantization output result. The shape of x is the same as that of y.

Constraints

rank, rankSize, and rankRoot must meet the following conditions:

0 ≤ rank < rankSize
0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```
Currently, multiple HCCP process instances cannot run on a single device, and multiple communication operators cannot run on a single device at the same time. For example, when a communication operator is running, multiple models cannot run on the same card.

Parent topic: atb/infer_op_params.h