Collective Communication

Broadcast

The Broadcast operation broadcasts the data of the root rank in the communicator to other ranks.

Note: Only one root rank is allowed in a communicator.

Related API: HcclBroadcast

Scatter

The Scatter operation evenly distributes data of the root rank in the communicator to other ranks.

Note: Only one root rank is allowed in a communicator.

Related API: HcclScatter

AllGather

The AllGather operation resorts the inputs of all ranks in a communicator by rank ID (in ascending order), combines the inputs, and sends the result to the output buffers of all ranks.

For the AllGather operation, each rank receives a set of data that is resorted by rank ID, that is, AllGather outputs of all ranks are the same.

Related API: HcclAllGather

AllGatherV

The AllGatherV operation resorts the inputs of all ranks in a communicator by rank ID (in ascending order), combines the inputs, and sends the result to the outputs of all ranks. Unlike the AllGather operation, AllGatherV allows different data sizes to be configured for the input of different ranks in the communicator.

For the AllGatherV operation, each rank receives a set of data that is resorted by rank ID, that is, AllGatherV outputs of all ranks are the same.

Related API: HcclAllGatherV

Reduce

The Reduce operation performs reduction (sum, prod, max, and min) over the input data of all ranks in a communicator, and then sends the result to the output buffer of the root rank.

Note: Only one root rank is allowed in a communicator.

Related API: HcclReduce

AllReduce

The AllReduce operation performs reduction (sum, prod, max, and min) over the input data of all ranks in a communicator, and then sends the result to the output buffers of all ranks.

Note: Each rank has only one input.

Related API: HcclAllReduce

ReduceScatter

The ReduceScatter operation evenly divides the input data of all ranks in a communicator into rank size parts, performs reduction (sum, prod, max, and min) on 1/rank size part of data of each rank, and distributes the result to the output buffer of each rank based on the number.

Related API: HcclReduceScatter

ReduceScatterV

The ReduceScatterV operation is similar to the ReduceScatter operation. The difference is that the ReduceScatterV operation supports the configuration of different data sizes for different ranks in a communicator. (The data size can be configured for different IDs in a rank, but the data size of the same ID in different ranks must be the same.) After the reduction operation (sum, prod, max, and min) is performed on data with the same ID in each rank, the result is distributed to the output buffer of each rank based on the ID.

Related API: HcclReduceScatterV

AlltoAll

The AlltoAll operation sends the same-sized data to all ranks in the communicator, and receives the same-sized data from all ranks.

The AlltoAll operation divides input data into blocks in a specific dimension, sends the blocks to other ranks in sequence, receives input data from other ranks, and concatenates data in a specific dimension in sequence.

Related API: HcclAlltoAll

AlltoAllV

The AlltoAllV operation sends data (with customizable size) to all ranks in the communicator and receives data from all ranks.

Related API: HcclAlltoAllV

AlltoAllVC

The AlltoAllVC operation sends data (with customizable size) to all ranks in the communicator and receives data from all ranks. Unlike AlltoAllV, AlltoAllVC uses the input parameter sendCountMatrix to pass the RX and TX parameters of all ranks.

Related API: HcclAlltoAllVC

API Call

The following uses the HcclAllReduce API as an example, whose prototype definition is as follows:

1

HcclResult HcclAllReduce(void *sendBuf, void *recvBuf, uint64_t count, HcclDataType dataType, HcclReduceOp op, HcclComm comm, aclrtStream stream)

HcclAllReduce performs the reduction operation on the input of all ranks within a communicator and broadcasts the result to the output of all ranks. The op parameter specifies the reduction operation type. HcclAllReduce allows only one input for each rank.

As shown in the following code snippet, all data elements in the input memory of the communicator are summed in the float32 data format (in the example, only one data element is involved in each rank), and then the sum result is sent to the output memory of all ranks.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

void* hostBuf = nullptr;
void* sendBuf = nullptr;
void* recvBuf = nullptr;
uint64_t count = 1;
int malloc_kSize = count * sizeof(float);
aclrtStream stream;
aclrtCreateStream(&stream);

// Allocate memory for collective communication.
aclrtMalloc((void**)&sendBuf, malloc_kSize, ACL_MEM_MALLOC_HUGE_ONLY); 
aclrtMalloc((void**)&recvBuf, malloc_kSize, ACL_MEM_MALLOC_HUGE_ONLY);

// Initialize the input memory.
aclrtMallocHost((void**)&hostBuf, malloc_kSize);
aclrtMemcpy((void*)sendBuf, malloc_kSize, (void*)hostBuf, malloc_kSize, ACL_MEMCPY_HOST_TO_DEVICE);

// Perform collective communication.
HcclAllReduce((void *)sendBuf, (void*)recvBuf, count, HCCL_DATA_TYPE_FP32, HCCL_REDUCE_SUM, hcclComm, stream);

For details about the complete call sample of the HcclAllReduce API, see the HcclAllReduce operation code samples of different communicator initialization modes in Sample Code.