HcclCommSymWinRegister

Applicability

Function

Registers the virtual memory as symmetric memory after allocating and mapping virtual and physical memory.

Symmetric memory is a memory management model that enables parallel processing units (such as ranks) to access each other's memory in a globally visible manner, without requiring explicit address exchange.

The following figure shows the basic implementation of symmetric memory, where each rank pre-reserves a virtual address space of identical size and layout.

• Allocate virtual memory for each rank. Assuming the virtual memory size per rank is heap_size and the number of ranks in the communicator is rank_size, the total virtual memory size across the communicator will be heap_size x rank_size.
• The virtual address layout for each rank is the same.
• Each rank's virtual addresses are mapped to the corresponding physical addresses, enabling direct access to the memory of other ranks.

Symmetric memory enables HCCL to operate directly on user-provided memory, completely bypassing intermediate buffers (such as the HCCL buffer) and eliminating memory copy overhead.

Prototype

1

HcclResult HcclCommSymWinRegister(HcclComm comm, void *addr, uint64_t size, CommSymWindow *winHandle, uint32_t flag)

Parameters

Returns

HcclResult: HCCL_SUCCESS on success, or else failure.

Constraints

• Only intra-SuperPoD communication of the Atlas A3 training product / Atlas A3 inference product is supported.
• The communication operator can be expanded only on AICPU.
• Only SDMA communication over HCCS between AI Servers within a SuperPoD is supported. RDMA communication over RoCE is not supported (that is, the environment variable HCCL_INTER_HCCS_DISABLE cannot be set to TRUE, and this environment variable is invalid in single-node scenarios).
• Only symmetric networking is supported, where each server has the same number of cards.
• This API supports only the collective communication operators AllGather, ReduceScatter, AllReduce, and AlltoAll.
• This registration API must be called collectively by all ranks within the communicator.
• The size of the physical memory mapped to each rank's input address must be identical, as symmetric memory registration is aligned with the physical memory size.
• The size parameter input by all ranks must be identical when calling this API.
• When using the symmetric memory function, the input and output memory of an operator must be registered as symmetric memory by calling this API.
• The memory registered by calling this API can be deregistered by calling the HcclCommSymWinDeregister API.

Call Example

// Create and initialize the configuration items of the communicator.
HcclCommConfig config;
HcclCommConfigInit(&config);
// Modify the communicator configuration as required.
config.hcclSymWinMaxMemSizePerRank = 10; // Set per-rank size in GB (Default: 16). Total reserved virtual symmetric heap size = rankSize x hcclSymWinMaxMemSizePerRank.

// Obtain the communicator parameters.
uint32_t rankSize = 4;
uint32_t rankId = 0;
int32_t deviceId;
ACLCHECK(aclrtGetDevice(&deviceId));
HcclRootInfo rootInfo;
HCCLCHECK(HcclGetRootInfo(&rootInfo));

// Initialize the collective communicator.
HcclComm hcclComm;
HCCLCHECK(HcclCommInitRootInfoConfig(rankSize, &rootInfo, rankId, &config, &hcclComm));

// Create a task flow.
aclrtStream stream;
ACLCHECK(aclrtCreateStream(&stream));

// Configure the physical memory attributes.
aclrtPhysicalMemProp prop;
prop.handleType = ACL_MEM_HANDLE_TYPE_NONE;
prop.allocationType = ACL_MEM_ALLOCATION_TYPE_PINNED;
prop.memAttr = ACL_HBM_MEM_HUGE;
prop.location.id = deviceId;
prop.location.type = ACL_MEM_LOCATION_TYPE_DEVICE;
prop.reserve = 0;

// Obtain the alignment granularity, which is usually 2 MB.
size_t granularity;
ACLCHECK(aclrtMemGetAllocationGranularity(&prop, ACL_RT_MEM_ALLOC_GRANULARITY_RECOMMENDED, &granularity));

// Align the size based on the granularity.
size_t size = 2 * 1024 * 1024;
size_t allocSize = (size + granularity - 1) / granularity * granularity;

// Reserve virtual memory.
void *virPtr;
ACLCHECK(aclrtReserveMemAddress(&virPtr, allocSize, 0, nullptr, 1));

// Allocate physical memory.
aclrtDrvMemHandle memHandle;
ACLCHECK(aclrtMallocPhysical(&memHandle, allocSize, &prop, 0));

// Establish the mapping from physical memory to virtual memory.
ACLCHECK(aclrtMapMem(virPtr, allocSize, 0, memHandle, 0));

size_t sendBytes = 1024;
size_t recvBytes = rankSize * sendBytes;
HcclCommSymWindow symWin;
// Register symmetric memory.
HCCLCHECK(HcclCommSymWinRegister(hcclComm, virPtr, sendBytes + recvBytes, &symWin, 1));

// Use symmetric memory.
void *sendBuff = virPtr;
void *recvBuff = static_cast<char*>(sendBuff) + sendBytes;

// Call the collective communication operator.
HCCLCHECK(HcclAllGather(sendBuff, recvBuff, sendBytes, HCCL_DATA_TYPE_INT8, hcclComm, stream));

// Wait until the collective communication task in the task flow is complete.
ACLCHECK(aclrtSynchronizeStream(stream));

// Deregister symmetric memory.
HCCLCHECK(HcclCommSymWinDeregister(symWin));

// Free memory.
ACLCHECK(aclrtUnmapMem(virPtr));
ACLCHECK(aclrtFreePhysical(memHandle));
ACLCHECK(aclrtReleaseMemAddress(virPtr));

 // Destroy the task flow.
ACLCHECK(aclrtDestroyStream(stream));

// Destroy the communicator.
HCCLCHECK(HcclCommDestroy(hcclComm));