Unilateral Communication

Key Advantages

• One-sided zero-copy communication: The unilateral communication library provides simple and reliable one-sided communication APIs. Once local memory data is ready, direct data transmission to remote memory can be completed through one-sided operations without any action required from the remote node. This provides core technical support for building scheduling mechanisms that overlap communication and computation. Meanwhile, the zero-copy capability enables direct data transmission between user memory spaces, avoiding redundant data movement, reducing memory bandwidth usage, and lowering memory capacity consumption.
• Hardware difference shielding and multi-link high-speed interconnection across devices: The unilateral communication library masks underlying hardware differences across Ascend series chips, so users do not need to adapt code for different chip architectures. At the communication link layer, it natively supports high-speed interconnection protocols such as RDMA and HCCS, with communication bandwidth up to 119 GB/s. It enables seamless high-speed interconnection across heterogeneous devices, meeting low-latency and high-throughput requirements.
• Minimalist API design and deep adaptation to open-source frameworks: Featuring a minimalist API design with only about 10 core calls, the unilateral communication library lowers the integration barrier for developers and provides C++ and Python language bindings. It has been deeply integrated with open-source frameworks such as Mooncake and DeepLink. Mainstream inference engines including vLLM and SGLang can directly invoke the unilateral communication library APIs to implement efficient cross-device KV cache transmission, reducing memory access latency by 20% during LLM inference and improving inference throughput.

Software Architecture

Figure 1 shows the unilateral communication library.

The core components are as follows:

• HIXL Engine: As the core transmission engine, it provides basic transmission APIs and supports multiple memory transmission types such as D2D, D2H, and H2D. It is compatible with various transmission protocols including HCCS and RDMA, enabling high-speed and reliable data transmission. It natively supports multiple data link types and is suitable for complex scenarios in homogeneous and heterogeneous clusters. It can efficiently adapt to link changes and schedule resources under dynamic cluster scaling, forming a reliable communication foundation for overall cluster operation.
• LLM-DataDist: Built on the HIXL Engine, it provides a set of data transmission APIs with KV cache semantics, enabling fast and flexible integration with inference engines such as vLLM and SGLang.

Quick Experience

The sample scenario is H2RD (data transmission from the host to the remote device), describing how the client unilaterally initiates data read and write operations to the server.

1. Bind the Ascend device used by the local process on both the client and server, initialize the HIXL object, allocate memory, and register with HIXL.
   • Client

     // Bind the Ascend device used by the local process.
     CHECK_ACL(aclrtSetDevice(device));
     
     // Initialize the HIXL engine with local_engine as the identifier. (Generally, the client uses an IP address string without a port number.)
     Hixl hixl_engine;
     std::map<AscendString, AscendString> options;
     CHECK_HIXL(hixl_engine.Initialize(local_engine, options));
     
     // Allocate the on-host transmission buffer for the local end of MEM_HOST registration and unilateral read/write.
     int32_t *host_buf = nullptr;
     CHECK_ACL(aclrtMallocHost(reinterpret_cast<void **>(&host_buf), sizeof(int32_t)));
     
     // Register the host address with HIXL so that the peer end can perform unilateral access through the established link.
     MemDesc reg_desc{};
     reg_desc.addr = reinterpret_cast<uintptr_t>(host_buf);
     reg_desc.len = sizeof(int32_t);
     MemHandle handle = nullptr;
     CHECK_HIXL(hixl_engine.RegisterMem(reg_desc, MEM_HOST, handle));

   • Server

     // Bind the Ascend device used by the local process.
     CHECK_ACL(aclrtSetDevice(device));
     
     // Initialize HIXL on the local_engine with a port as the connected end.
     Hixl hixl_engine;
     std::map<AscendString, AscendString> options;
     CHECK_HIXL(hixl_engine.Initialize(local_engine, options));
     
     // Allocate memory on the device and write the initial value 1 through H2D for the client to verify read.
     constexpr int32_t kInit = 1;
     int32_t *dev_buf = nullptr;
     CHECK_ACL(aclrtMalloc(reinterpret_cast<void **>(&dev_buf), sizeof(int32_t), ACL_MEM_MALLOC_HUGE_ONLY));
     CHECK_ACL(aclrtMemcpy(dev_buf, sizeof(int32_t), &kInit, sizeof(int32_t), ACL_MEMCPY_HOST_TO_DEVICE));
     
     // Register the device address with HIXL so that the client can read and write the address range unilaterally.
     MemDesc reg_desc{};
     reg_desc.addr = reinterpret_cast<uintptr_t>(dev_buf);
     reg_desc.len = sizeof(int32_t);
     MemHandle handle = nullptr;
     CHECK_HIXL(hixl_engine.RegisterMem(reg_desc, MEM_DEVICE, handle));

2. Initiate a link establishment request from the client to the server.

   // Establish a transmission link with remote_engine (server listening address).
   CHECK_HIXL(hixl_engine.Connect(remote_engine));

3. Initiate data read/write from the client to the server, and check whether the read/write values are normal on both the client and server.
   • Client

     // Unilateral read: Read the pre-written 1 from the MEM_DEVICE of the peer server to the local host memory.
     TransferOpDesc xfer{reinterpret_cast<uintptr_t>(&host_buf), dev_buf, sizeof(int32_t)};
     CHECK_HIXL(hixl_engine.TransferSync(remote_engine, READ, {xfer}));
     printf("after READ: %d\n", *host_buf);
     
     // Unilateral write: Write the locally preset host memory 2 to the MEM_DEVICE of the peer server.
     *host_buf = 2;
     CHECK_HIXL(hixl_engine.TransferSync(remote_engine, WRITE, {xfer}));

   • Server

     // After the client completes the unilateral write, the server reads the value from the device to the host. The value should be 2 after the client performs the write operation.
     int32_t host_val = 0;
     CHECK_ACL(aclrtMemcpy(&host_val, sizeof(int32_t), dev_buf, sizeof(int32_t), ACL_MEMCPY_DEVICE_TO_HOST));
     printf("after transfer: %d\n", host_val);

4. Release the link on the client.

   CHECK_HIXL(hixl_engine.Disconnect(remote_engine));

5. On both the client and server, deregister the memory, free the memory, destroy the HIXL engine, and unbind the engine from the device.
   • Client

     CHECK_HIXL(hixl_engine.DeregisterMem(handle));
     CHECK_ACL(aclrtFreeHost(host_buf));
     hixl_engine.Finalize();
     CHECK_ACL(aclrtResetDevice(device));

   • Server

     CHECK_HIXL(hixl_engine.DeregisterMem(handle));
     CHECK_ACL(aclrtFree(dev_buf));
     hixl_engine.Finalize();
     CHECK_ACL(aclrtResetDevice(device));