Memory Management

Function Description

Memory management involves the following APIs and functions.

API	Function
RegisterMem	Registers memory. This API must be called by the client and server before link establishment.
DeregisterMem	Deregisters memory. This API must be called by the client and server after link disconnection.
TransferSync	Reads memory from a remote address to the local side, or pushes local memory to the memory corresponding to a remote address.

Scenario

This capability is mainly used for memory transmission between distributed clusters.

Example

This example demonstrates API usage, covering memory registration, deregistration, and transmission. Pseudocode examples are provided below for each role.

Initialize the HIXL on the client and server as described in Link Establishment.

Allocate memory of the required size for each request on both the client and server sides and register it. If the operation fails, release the corresponding resources. This must be performed before link establishment.

void OnError(Hixl &engine, const std::vector<MemHandle> &handles)
{
    for (auto handle : handles) {
        (void) engine.DeregisterMem(handle);
    }
    engine.Finalize();
}

MemDesc desc{};
desc.addr = reinterpret_cast<uintptr_t>(addr);
desc.len = len;
MemHandle handle = nullptr;
auto ret = engine.RegisterMem(desc, MEM_HOST, handle);
if (ret != SUCCESS) {
    printf("[ERROR] RegisterMem failed, ret = %u\n", ret);
    return -1;
}

Transmit memory using either of the following methods.

In scenarios where Atlas A3 training product/Atlas A3 inference product uses HCCS for D2RH or RH2D transmission, you can enable the OPTION_ENABLE_USE_FABRIC_MEM configuration item to achieve optimal transmission performance.

On the client, call TransferSync with the operation parameter set to READ to read memory from a remote address to the local side. The local and remote addresses must be registered locally and remotely before link establishment.

TransferOpDesc desc{reinterpret_cast<uintptr_t>(&local), reinterpret_cast<uintptr_t>(remote), size};
auto ret = engine.TransferSync(remote_engine, READ, {desc});
if (ret != SUCCESS) {
    printf("[ERROR] TransferSync read failed, ret = %u\n", ret);
    return -1;
}

On the client, call TransferSync with the operation parameter set to WRITE to push memory from a local address to the remote side. The local and remote addresses must be registered locally and remotely before link establishment.

TransferOpDesc desc{reinterpret_cast<uintptr_t>(local), reinterpret_cast<uintptr_t>(remote), size};
auto ret = engine.TransferSync(remote_engine, WRITE, {desc});
if (ret != SUCCESS) {
    printf("[ERROR] TransferSync write failed, ret = %u\n", ret);
    return -1;
}

When the service exits, disconnect the links on Prefill and Decode sides as described in Link Disconnection and call finalize to release resources.

Exception Handling

TIMEOUT occurs when a transmission operation times out, indicating a link issue. To address this, try disconnecting and re-establishing the link.
NOT_CONNECTED occurs when no link has been established with the remote side.

For more troubleshooting methods, see HIXL Error Codes.

Parent topic: Function Description