HcclBatchSendRecv

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	☓
Atlas inference product	☓
Atlas training product	√

For Atlas A2 training product/Atlas A2 inference product, only the Atlas 800T A2 training server, Atlas 900 A2 PoD cluster basic unit, and Atlas 200T A2 Box16 heterogeneous subrack are supported.

Function

Completes sending and receiving tasks in batches on the current rank. The sending and receiving tasks of the current rank are asynchronous and do not block each other.

Prototype

HcclResult HcclBatchSendRecv(HcclSendRecvItem* sendRecvInfo, uint32_t itemNum, HcclComm comm, aclrtStream stream)

Parameters

Parameter	Input/Output	Description
sendRecvInfo	Input	Start address of the list of sending and receiving tasks to be distributed in the rank. HcclSendRecvItem type. For details, see HcclSendRecvItem. Note that the int128 data type is not supported by the Atlas 350 Accelerator Card.
itemNum	Input	Number of tasks to be received and sent by the rank.
comm	Input	Communicator where the operation is performed.
stream	Input	Stream of the rank.

Returns

HcclResult: HCCL_SUCCESS on success, or else failure.

Constraints

"Asynchronous" means that the sending and receiving tasks on the same device are asynchronous and do not block each other. However, the sending and receiving tasks between devices are still synchronous. Therefore, the sending and receiving tasks between devices must be in one-to-one mapping, which is the same as HcclSend and HcclRecv.
For the Atlas A2 training product/Atlas A2 inference product, when this API is used in a large-scale cluster (rankSize > 500), the number of concurrent executions cannot exceed 3.
For the Atlas 200T A2 Box16 heterogeneous subrack, if a link fails to be set up between devices in the server (error code: EI0010), set HCCL_INTRA_ROCE_ENABLE to 1 and HCCL_INTRA_PCIE_ENABLE to 0 to enable the communication between the devices in the server through the RoCE loop. (Ensure that the server has RoCE NICs and the RDMA links between the devices that can send and receive data are connected.) The following is an example of configuring the environment variables:
```
export HCCL_INTRA_ROCE_ENABLE=1
export HCCL_INTRA_PCIE_ENABLE=0
```

Call Example

// Allocate device memory for collective communication.
void *sendBuf = nullptr;
void *recvBuf = nullptr;
uint64_t count = 8;
size_t mallocSize = count * sizeof(float);
aclrtMalloc((void **)&sendBuf, mallocSize, ACL_MEM_MALLOC_HUGE_ONLY);
aclrtMalloc((void **)&recvBuf, mallocSize, ACL_MEM_MALLOC_HUGE_ONLY);

// Initialize the communicator.
uint32_t rankSize = 8;
HcclComm hcclComm;
HcclCommInitRootInfo(rankSize, &rootInfo, deviceId, &hcclComm);

// Create a task flow.
aclrtStream stream;
aclrtCreateStream(&stream);

// Perform the Send/Recv operation to send data to the next rank and receive data from the previous rank.
// HcclBatchSendRecv can deliver multiple RX and TX tasks on the local rank at the same time.
uint32_t next = (deviceId + 1) % count;
uint32_t prev = (deviceId - 1 + count) % count;
HcclSendRecvItem sendRecvInfo[2];
sendRecvInfo[0] = HcclSendRecvItem{HCCL_SEND, sendBuf, count, HCCL_DATA_TYPE_FP32, next};
sendRecvInfo[1] = HcclSendRecvItem{HCCL_RECV, recvBuf, count, HCCL_DATA_TYPE_FP32, prev};
HcclBatchSendRecv(sendRecvInfo, 2, hcclComm, stream);

// Wait until the collective communication task in the task flow is complete.
ACLCHECK(aclrtSynchronizeStream(stream));

// Free resources.
aclrtFree(sendBuf);          // Free the device memory.
aclrtFree(recvBuf);          // Free the device memory.
aclrtDestroyStream(stream);  // Destroy the task flow.
HcclCommDestroy(hcclComm);   // Destroy the communicator.

Parent topic: Point-to-Point Communication