AllToAllOperation

Applicable Products

Hardware Model	Supported
Atlas 350 accelerator cards	x
Atlas A3 inference products / Atlas A3 training products	√
Atlas A2 training products / Atlas A2 inference products	√
Atlas training products	x
Atlas inference products	x
Atlas 200I/500 A2 inference products	x

Description

Sends data of the same amount (the input is divided into rankSize parts) to all devices in the communicator, and receives data of the same amount from all devices. As shown in the following figure, when rankSize=4, each device evenly divides its data into rankSize parts and sends the data to other devices.

Figure 1 Functions of the AllToAll operator

Computational process (Python):

      
           goldenTensors = []
for i in range(rankSize):
    golden_out = []
    for j in range(rankSize):
        golden_out_list = intensors[j].reshape(-1).tolist() # intensors[j] indicates the intensor on the jth device, which is considered as a one-dimensional vector during computation.
        split = golden_out_list[i*len(golden_out_list) // rankSize:(i+1)*len(golden_out_list) // rankSize]
        golden_out += split
    golden_out_tensor = torch.tensor(golden_out).reshape(shape) # golden_out_tensor is the output tensor of the ith device. The output shape is the same as the input tensor.
    goldenTensors.append(golden_out_tensor)
return [goldenTensors[rank].cpu()]

Operator Context

Figure 2 Position of AllToAll in communication INT8 quantization

Input the quantized tensor X of int8 and send it to each node through AllToAll. (In this example, int8 is used for communication, which improves the communication speed.)

Then, the sum operation of reduce is used to sum up X, the int8 data is dequantized to the float16 data, and AllGather is used for communication to transmit the calculation result to each node.

Application Scenarios

Send data to each node in many-to-many mode. AllToAll is an extension of AllGather. Compared with AllGather, different AllToAll nodes collect different data from a node.

Figure 3 Comparison between AllToAll and AllGather

It applies to model parallelism, matrix transposition in model parallelism, and matrix transposition from data parallelism to model parallelism.

Definition

      
           struct AllToAllParam {
    int rank = 0;
    int rankSize = 0;
    int rankRoot = 0;
    std::string backend = "hccl";
    HcclComm hcclComm = nullptr;
    CommMode commMode = COMM_MULTI_PROCESS;
    std::string rankTableFile;
    std::string commDomain;
    bool transpose = false;
    uint8_t rsv[62] = {0};
};

Parameters

Member	Type	Default Value	Description
rank	int	0	Rank of the current device within the communicator.
rankSize	int	0	Number of devices.
rankRoot	int	0	Root rank for the operation.
backend	std::string	hccl	Communication computation type. hccl and lccl are supported. The LCCL function will be deleted in later versions. You are advised to use the HCCL function. If backend is set to lccl, for Atlas 200T A2 Box16 heterogeneous subrack, the communicator can be created only among the first eight or last eight devices. A communicator that contains both the first eight and last eight devices is not supported.
hcclComm	HcclComm	nullptr	Pointer to the HCCL communicator. By default, this parameter is left blank. The ATB is created by users. If the user wants to manage the communicator, the communicator pointer needs to be passed. The ATB uses the passed communicator pointer to execute the communication operator.
commMode	CommMode	COMM_MULTI_PROCESS	Communication mode. For details, see CommMode. In the scenario where hccl multi-thread is applied, a communicator can only be imported externally.
rankTableFile	std::string	-	Path of the configuration file for cluster information, which applies to single-node and multi-node communication scenarios. Currently, only the HCCL backend is supported. If a rank table is configured for a single-node system, the rank table is used to initialize the communicator.
commDomain	std::string	-	Communicator name used by a communication device group when there are multiple communicators. When backend is set to lccl and commMode is multi-process, commDomain must be set to a number ranging from 0 to 65535. When commMode is multi-thread, deterministic computing is not supported. In this case, LCCL_DETERMINISTIC must be set to 0 or false. For multi-process/multi-thread multi-communicator concurrency, LCCL_PARALLEL must be set to 1 or true. If multi-communicator parallelism is no longer needed, set LCCL_PARALLEL to 0 or false to prevent performance deterioration in basic scenarios. The memory size of the communicator can be set for performance optimization. The format is commDomain:commDomain size (MB). For example, 0:400. When the configurations of multiple operators in the same communicator conflict, the configuration of the first operator executed in the communicator is used. Restriction: The default value is 200. Currently, the value cannot be less than 200.
transpose	bool	false	Transposes the input based on the communication result. Currently, only lccl is supported.
rsv[62]	uint8_t	{0}	Reserved

Input/Output in Basic Scenarios

transpose is set to false.

Parameter	Dimension	Data Type	Format	Description
x	[dim_0, dim_1, ..., dim_n]	float16/float/int8/int16/int32/int64/bf16	ND	Input tensor
output	[dim_0, dim_1, ..., dim_n]	float16/float/int8/int16/int32/int64/bf16 The data type is the same as that of the input.	ND	Output tensor with the same dimensions as the input.

Parameter

Dimension

Data Type

Format

Description

[dim_0, dim_1, ..., dim_n]

float16/float/int8/int16/int32/int64/bf16

Input tensor

output

[dim_0, dim_1, ..., dim_n]

float16/float/int8/int16/int32/int64/bf16

The data type is the same as that of the input.

Output tensor with the same dimensions as the input.

Input/Output in Transpose Scenarios

transpose is set to true.

Parameter	Dimension	Data Type	Format	Description
x	[b, w]	float16/bf16/int8	ND	Input tensor.
output	[b*rankSize, w/rankSize]	float16/bf16/int8	ND	Output tensor.

Constraints

When the backend is lccl, multi-node communication and multi-communicators are not supported, and only even rankSize is supported.
The input tensor dimensions of each communication card are the same.
The product of the parameter dimensions must be exactly divided by rankSize.
During AllToAll computation, the input tensor is considered as a 1D vector, and the output shape is the same as that of the input tensor.
The constraints on transposition are:
- w must be exactly divided by rankSize.
- w/rankSize cannot exceed 90 KB.
- The size of the x tensor cannot exceed 190 MB.
- The value of rankSize can only be 2, 4, or 8.
rank, rankSize, and rankRoot must meet the following conditions:
- 0 ≤ rank < rankSize
- 0 ≤ rankRoot < rankSize

If there are multiple users, ATB_SHARE_MEMORY_NAME_SUFFIX (see ATB Environment Variables) needs to be used to distinguish the shared memory and synchronize initialization information.
When the communication operator of the ATB exits abnormally, run the following commands to clear the residual data to avoid affecting subsequent operations.
```
rm -rf /dev/shm/sem.lccl*
rm -rf /dev/shm/sem.hccl*
ipcrm -a
```

Parent topic: atb/infer_op_params.h