AllToAllVOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	x
Atlas A3 inference products / Atlas A3 training products	√
Atlas A2 training products / Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Sends data to all devices in the communicator (the data volume can be customized using parameters) and receives data from all devices (the data volume can be customized using parameters).

Example
Figure 1 Operator input and output example

Python computation process

        
             golden_result = []
for j in range(rankSize):
    tensor_list = intensors[j].reshape(-1).tolist() # intensors[j] stands for the intensor on the jth device.
    golden_result += tensor_list[sdispls[rank]:sdispls[rank]+sendCounts[rank]]
size = sum(recvCounts)
golden_out_tensor = torch.tensor(golden_result).reshape(1,size)

Operator Context

Figure 2 Position of AllToAll in communication INT8 quantization

Input the quantized tensor X of int8 and send it to each node through AllToAll. (In this example, int8 is used for communication, which improves the communication speed.)

Then, the sum operation of reduce is used to sum up x, the int8 data is dequantized to the float16 data, and AllGather is used for communication to transmit the calculation result to each node.

Application Scenarios

Send data to each node in many-to-many mode. It is an extension of AllGather, but different nodes collect different data from a node.

Figure 3 Comparison between AllToAll and AllGather

It applies to model parallelism, matrix transposition in model parallelism, and matrix transposition from data parallelism to model parallelism.

Definition

      
           struct AllToAllVParam {
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::vector<int64_t> sendCounts;
    std::vector<int64_t> sdispls;
    std::vector<int64_t> recvCounts;
    std::vector<int64_t> rdispls;
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    uint8_t rsv[64] = {0};
};

Parameters

Member	Type	Default Value	Description
rank	int	0	Rank of the current device within the communicator.
rankSize	int	0	Number of devices.
rankRoot	int	0	Root rank for the operation.
sendCounts	std::vector< int64_t >	[]	Array of the sent data volume. For example, if the data type is float32, sendCounts[i] = n indicates that the current rank sends n float32 data segments to rank i.
sdispls	std::vector< int64_t >	[]	Array of sending offsets. sdispls[i] = n indicates that the current rank sends data to rank i from a position whose offset relative to the input start position is n.
recvCounts	std::vector< int64_t >	[]	Array of the received data volume. For example, if the data type is float32, recvCounts[i] = n indicates that the current rank receives n float32 data from rank i.
rdispls	std::vector< int64_t >	[]	Array of receive offsets. rdispls[i] = n indicates that the current rank starts to receive data of rank i from a position whose offset relative to the output start position is n.
backend	std::string	hccl	Communication computation type. Only hccl is supported.
hcclComm	HcclComm	nullptr	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	Communication mode. For details, see CommMode. In the scenario where hccl multi-thread is applied, a communicator can only be imported externally.
rankTableFile	std::string	-	Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported. If a rank table is configured for a single-node system, the rank table is used to initialize the communicator.
commDomain	std::string	-	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain must be set to a number ranging from 0 to 65535. When commMode is multi-thread, deterministic computing is not supported. In this case, LCCL_DETERMINISTIC must be set to 0 or false. For multi-process/multi-thread multi-communicator concurrency, LCCL_PARALLEL must be set to 1 or true. If multi-communicator parallelism is no longer needed, set LCCL_PARALLEL to 0 or false to prevent performance deterioration in basic scenarios. The memory size of the communicator can be set for performance optimization. The format is commDomain:commDomain size (MB). For example, 0:400. When the configurations of multiple operators in the same communicator conflict, the configuration of the first operator executed in the communicator is used. Restriction: The default value is 200. Currently, the value cannot be less than 200.
rsv[64]	uint8_t	{0}	Reserved

Input

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	"hccl": float16/float/int8/int16/int32/int64/bf16	ND	Input tensor

Output

Parameter	Dimension	Data Type	Format	Description
output	[1,recvCountsSum] recvCountsSum is the sum of all elements of the recvCounts parameter.	"hccl": float16/float/int8/int16/int32/int64/bf16 The data type is the same as the input.	ND	Output tensor. The shape of the last dimension is the sum of all elements of the recvCounts parameter.

Parameter

Dimension

Data Type

Format

Description

output

[1,recvCountsSum]

recvCountsSum is the sum of all elements of the recvCounts parameter.

"hccl": float16/float/int8/int16/int32/int64/bf16

The data type is the same as the input.

Output tensor. The shape of the last dimension is the sum of all elements of the recvCounts parameter.

Constraints

The length of the array parameters sendCounts, recvCounts, sdispls and rdispls is equal to the value of rankSize, and the values of all elements in the array are greater than 0.
The input tensor dimensions of each communication card are the same.
The sum of sendCounts and recvCounts array elements cannot overflow int64, and the sum of recvCounts arrays must be greater than 0. For any element in rdispls, recvCounts[i] + rdispls[i] cannot be greater than recvCountsSum (the last dimension of the output). For any element in sdispls, sendCounts[i] + sdispls[i] cannot be greater than the data volume of the input tensor. For example, if the input tensor has shape [3, 4, 5] and the data amount is 60 (3 x 4 x 5), the input tensor will be considered as a tensor with shape [60] for AllToAllV computation.
rank, rankSize, and rankRoot must meet the following conditions:
- 0 ≤ rank < rankSize
- 0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```

Parent topic: atb/infer_op_params.h