ReduceScatterOperation

Applicable Products

Hardware Model	Supported or Not	Remarks
Atlas 350 accelerator card	x	-
Atlas A3 inference products/Atlas A3 training products	√	-
Atlas A2 training products/Atlas A2 inference products	√	Not supported by Atlas A2 inference products.
Atlas training products	√	-
Atlas inference products	√	-
Atlas 200I/500 A2 inference products	x	-

Description

Many-to-many: The same Reduce operation is performed on all nodes in the cluster by dimension, and then the result is distributed to all nodes in the cluster. This operator does not support deterministic computing.

Figure 1 Communication principle

Note: The hardware type supports only the Atlas A2 training products/Atlas A2 inference products and Atlas A3 inference products/Atlas A3 training products, and does not support deterministic computing and multi-node communication.

Application Scenarios

Data parallelism and model parallelism.
ReduceScatter in the combination of ReduceScatter and Allgather in allReduce in data parallelism.
ReduceScatter in the backward computation after forward all-gather in model parallelism.

Examples

Example: rankSize=2

reduceType=sum

Shape of the input tensor: (2 * 3 * 4)

Input tensor:

input_tensor_0 = [[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]],
                  [[13, 14, 15, 16], [17, 18, 19, 20], [21, 22, 23, 24]]]
input_tensor_1 = [[[25, 26, 27, 28], [29, 30, 31, 32], [33, 34, 35, 36]],
                  [[37, 38, 39, 40], [41, 42, 43, 44], [45, 46, 47, 48]]]

Shape of the output tensor: (1 * 3 * 4)

Output tensor:

output_tensor_0 = [[[26, 28, 30, 32],[34, 36, 38, 40],[42, 44, 46, 48]]]
output_tensor_1 = [[[50, 52, 54, 56],[58, 60, 62, 64],[66, 68, 70, 72]]]

Definition

struct ReduceScatterParam{
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string reduceType = "sum";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string backend = "lccl";
    std::string rankTableFile;
    std::string commDomain;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Value	Mandatory or Not	Description
rank	int	0	[0, rankSize-1]	Yes	Communication ID of the current card.
rankSize	int	0	-	Yes	Number of communication cards.
rankRoot	int	0	[0, rankSize-1]	Yes	Primary communication ID.
reduceType	string	"sum"	sum prod max min	Yes	Communication computation type. The value can be sum, prod, max, or min.
backend	string	"lccl"	lccl/hccl	Yes	Communication computation type. Only hccl and lccl are supported. The LCCL function will be deleted in later versions. You are advised to use the HCCL function. When backend is set to hccl: sum, prod, max, and min are supported. On the Atlas A2 training products/Atlas A2 inference products, if reduceType is set to prod, the data type cannot be int16 or bf16. The Atlas training products do not support int16 or bf16. When backend is set to lccl: sum, max, and min are supported. In the Atlas 200T A2 Box16 heterogeneous subrack scenario, the communicator can be created only among the first eight or last eight cards. A communicator that contains both the first eight and last eight cards is not supported.
hcclComm	HcclComm	nullptr	-	No	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	COMM_MULTI_PROCESS/COMM_MULTI_THREAD	No	Communication mode. For details, see "CommMode."
rankTableFile	string	None	-	No	Path of the configuration file for cluster information.
commDomain	string	None	-	No	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain needs to be set to a value ranging from 0 to 65535. The memory size of the communicator can be set for performance optimization. The configuration format is commDomain:commDomain size (MB), for example, 0:400. If the configurations of multiple operators in the same communicator conflict, the configuration of the first executed operator in the communicator is used. Restriction: The default value is 200. Currently, the value cannot be less than 200.
rsv[64]	uint8_t	{0}	[0]	No	Reserved parameter.

Input

Parameter	Dimension	Data Type	Format	Description
x	[rankSize*n, dim_0, dim_1, ..., dim_n]	"lccl": float16/float/int8/int16/int32/bf16 "hccl": float16/float/int8/int16/int32/bf16	ND	Input tensor. The dimension is less than or equal to 8. The size of the first dimension is n times the value of rankSize (n is a positive integer).

Parameter

Dimension

Data Type

Format

Description

[rankSize*n, dim_0, dim_1, ..., dim_n]

"lccl": float16/float/int8/int16/int32/bf16

"hccl": float16/float/int8/int16/int32/bf16

Input tensor. The dimension is less than or equal to 8. The size of the first dimension is n times the value of rankSize (n is a positive integer).

Output

Parameter	Dimension	Data Type	Format	Description
output	[n, dim_0, dim_1, ..., dim_n]	"lccl": float16/float/int8/int16/int32/bf16 "hccl": float16/float/int8/int16/int32/bf16	ND	Output tensor. The dimension is less than or equal to 8. The size of the first dimension is n=x[0]/rankSize. From the second dimension, the output dimension is the same as the dimension and dimension value of input x, and the data types are also the same.

Parameter

Dimension

Data Type

Format

Description

output

[n, dim_0, dim_1, ..., dim_n]

"lccl": float16/float/int8/int16/int32/bf16

"hccl": float16/float/int8/int16/int32/bf16

Output tensor. The dimension is less than or equal to 8. The size of the first dimension is n=x[0]/rankSize. From the second dimension, the output dimension is the same as the dimension and dimension value of input x, and the data types are also the same.

Restrictions

rank, rankSize, and rankRoot must meet the following conditions:
- 0 ≤ rank < rankSize
- 0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```
Currently, multiple HCCP process instances cannot run on a single device, and multiple communication operators cannot run on a single device at the same time. For example, when a communication operator is running, multiple models cannot run on the same card.

Parent topic: atb/infer_op_params.h