Hierarchical Communication Principles

The following uses communication operators ReduceScatter, AllGather, and AllReduce as examples to describe the hierarchical communication process.

ReduceScatter

As shown in Figure 1, the ReduceScatter operator requires the ith rank to obtain the ith reduction result. To ensure the continuity of data chunks between servers, the ReduceScatter operation is first executed across servers and then within each server.

Figure 1 Hierarchical communication process of the ReduceScatter operator

AllGather

As shown in Figure 2, the AllGather operator requires that the input data of the ith rank be in the ith position of the result. To ensure the continuity of data chunks between servers, the AllGather operation is first executed within a server and then across servers.

Figure 2 Hierarchical communication process of the AllGather operator

AllReduce

As shown in Figure 3, the output of the AllReduce operator is a complete reduction result. Although the output is divided into two phases—ReduceScatter and AllGather, their semantics do not need to be strictly followed, and the communication process with large-sized data can be performed within a server with higher bandwidth. That is, the ReduceScatter operation is first performed within each server, then across servers, and finally within each server.

Figure 3 Hierarchical communication process of the AllReduce operator