AllGatherOperation

Applicable Products

Product	Supported
Atlas 350 accelerator cards	√
Atlas A3 inference products/Atlas A3 training products	√
Atlas A2 training products/Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	√
Atlas 200I/500 A2 inference products	x

Description

Aggregates data of a plurality of communications cards in the first dimension according to rank sequences and sends data to each card.

This operator involves multi-device operations. You can configure HCCL environment variables as required. For details, see Collective Communication in CANN Environment Variables.

Application Scenarios

Many-to-many: The data of multiple nodes is collected to a master node (Gather), and then the collected data is distributed to other nodes (broadcast).

Figure 1 Communication principle

Application scenarios:

AllGather can be used for model parallelism.
In model parallelism, parameters in forward computation are fully synchronized. Forward computation can be performed only after AllGather is used to synchronize the parameters split to different NPUs to one NPU.

Examples

Parameter setting
rankSize=2

Input and output

npu0 (primary device):

[[16, 56],
[ 5, 96]]

output=

[[[ 16,  56],
  [  5,  96]],

  [[ -7, -91],
  [-75,  38]]]

npu1:

[[ -7, -91],
[-75,  38]]

output=

[[[ 16,  56],
 [  5,  96]],
[[ -7, -91],
[-75,  38]]]

Definition

struct AllGatherParam {
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Description
rank	int	0	Rank of the current device within the communicator.
rankSize	int	0	Number of devices in the communicator.
rankRoot	int	0	Root rank for the operation.
backend	string	hccl	Communication backend. Only hccl and lccl are supported. The LCCL function will be deleted in later versions. You are advised to use the HCCL function. For Atlas inference products and Atlas 350 accelerator cards, backend can only be hccl. When backend is lccl and the machine is the Atlas 200T A2 Box16 in Atlas A2 training products, 16 devices can be deployed on a single server (full topology communication of 16 devices or communication of any device on a single node). Note: This is an experimental function. You are advised not to use it.
hcclComm	HcclComm	nullptr	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	Communication mode. For details, see CommMode. When HCCL multi-threading is applied, a communicator can only be imported externally.
rankTableFile	string	-	Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported. If a rank table is configured for a single-node system, the rank table is used to initialize the communicator. For details, see 《HCCL集合通信库》中的“ranktable文件配置资源信息”章节.
commDomain	string	-	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain needs to be set to a value ranging from 0 to 65535. The memory size of the communicator can be set for performance optimization. The configuration format is commDomain:commDomain size (MB), for example, 0:400. If the configurations of multiple operators in the same communicator conflict, the configuration of the first executed operator in the communicator is used. Restriction: The default value is 200. Currently, the value cannot be less than 200.
rsv[64]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	If backend is set to hccl: float16/float/int8/int16/int32/int64/bf16 If backend is set to lccl: float16/float/int8/int16/int32/int64/bf16	ND	Input tensor with less than 8 dimensions.

Output

Parameter	Dimension	Data Type	Format	Description
output	[rankSize, dim_0, dim_1, ... , dim_n]	If backend is set to hccl: float16/float/int8/int16/int32/int64/bf16 If backend is set to lccl: float16/float/int8/int16/int32/int64/bf16	ND	Output tensor with 8 dimensions or less. The number of dimensions of output is one dimension more than that of x.

Parameter

Dimension

Data Type

Format

Description

output

[rankSize, dim_0, dim_1, ... , dim_n]

If backend is set to hccl: float16/float/int8/int16/int32/int64/bf16
If backend is set to lccl: float16/float/int8/int16/int32/int64/bf16

Output tensor with 8 dimensions or less.

The number of dimensions of output is one dimension more than that of x.

Constraints

rank, rankSize, and rankRoot must meet the following conditions:
- 0 ≤ rank < rankSize
- 0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```
Currently, multiple HCCP process instances cannot run on a single device, and multiple communication operators cannot run on a single device at the same time. For example, when a communication operator is running, multiple models cannot run on the same card.

Parent topic: atb/infer_op_params.h