AllGatherOperation

Description

Aggregates data of a plurality of communications cards in the first dimension according to rank sequences and sends data to each card.

This operator involves multi-device operations. You can configure HCCL environment variables as required. For details, see . For details about how to configure TLS, see the HCCN Tool API Reference of the corresponding device.

Application Scenarios

Many-to-many: The data of multiple nodes is collected to a master node (Gather), and then the collected data is distributed to other nodes (broadcast).

Figure 1 Communication principle

Application scenarios:

AllGather can be used for model parallelism.
In model parallelism, parameters in forward computation are fully synchronized. Forward computation can be performed only after AllGather is used to synchronize the parameters split to different NPUs to one NPU.

Examples

Parameter setting
rankSize=2

Input and output

npu0 (primary communication card):

[[16, 56],
[ 5, 96]]

output=

[[[ 16,  56],
  [  5,  96]],

  [[ -7, -91],
  [-75,  38]]]

npu1:

[[ -7, -91],
[-75,  38]]

output=

[[[ 16,  56],
 [  5,  96]],
[[ -7, -91],
[-75,  38]]]

Definition

struct AllGatherParam {
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Description
rank	int	0	Communication ID of the current card
rankSize	int	0	Number of communication cards
rankRoot	int	0	Primary communication rank.
backend	string	"hccl"	Communication backend. Only hccl and lccl are supported. For the Atlas inference products, backend can only be hccl. When backend is set to lccl and the host topology is the single-server 16-card topology with the Atlas A2 inference products, only full topology communication of 16 cards or communication of any card on a single node is supported.
hcclComm	HcclComm	nullptr	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	Communication mode. For details, see CommMode. In the scenario where hccl multi-thread is applied, a communicator can only be imported externally.
rankTableFile	string	-	Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported. If a ranktable is configured for a single-node system, the ranktable is used to initialize the communicator. For details, see .
commDomain	string	-	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain must be set to a number ranging from 0 to 63. When commMode is multi-thread, deterministic computing is not supported. In this case, LCCL_DETERMINISTIC must be set to 0 or false. In the multi-process/multi-thread multi-communicator parallelism scenario, LCCL_PARALLEL must be set to 1 or true. After the multi-communicator parallelism is used, LCCL_PARALLEL must be set to 0 or false. Otherwise, the performance in basic scenarios deteriorates.
rsv[64]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[-1,…,-1] The value -1 indicates that the size of the current dimension is not restricted.	"hccl": float16/float/int8/int16/int32/int64/bf16 "lccl": float16/float/int8/int16/int32/int64/bf16	ND	Input tensor with less than 8 dimensions.

Parameter

Dimension

Data Type

Format

Description

[-1,…,-1]

The value -1 indicates that the size of the current dimension is not restricted.

"hccl": float16/float/int8/int16/int32/int64/bf16
"lccl": float16/float/int8/int16/int32/int64/bf16

Input tensor with less than 8 dimensions.

Output

Parameter	Dimension	Data Type	Format	Description
output	[rankSize, -1,…,-1] The value -1 indicates that the size of the current dimension is not restricted.	"hccl": float16/float/int8/int16/int32/int64/bf16 "lccl": float16/float/int8/int16/int32/int64/bf16	ND	Output tensor with 8 dimensions or less. The number of dimensions of output is one dimension more than that of x.

Parameter

Dimension

Data Type

Format

Description

output

[rankSize, -1,…,-1]

The value -1 indicates that the size of the current dimension is not restricted.

"hccl": float16/float/int8/int16/int32/int64/bf16
"lccl": float16/float/int8/int16/int32/int64/bf16

Output tensor with 8 dimensions or less.

The number of dimensions of output is one dimension more than that of x.

Restrictions

rank, rankSize, and rankRoot must meet the following conditions:
- 0 ≤ rank < rankSize
- 0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```
Currently, multiple HCCP process instances cannot run on a single device, and multiple communication operators cannot run on a single device at the same time. For example, when a communication operator is running, multiple models cannot run on the same card.

Parent topic: atb/infer_op_params.h