Unilateral communication

Core Advantages

• One-Sided Zero-Copy communication mechanism: The One-Sided Zero-Copy provides a simple and reliable one-sided communication interface. After the data in the local memory is ready, the data can be directly transmitted to the remote memory through one-sided operations. This provides core technical support for building scheduling mechanisms that overlap communication and computation. Meanwhile, the zero-copy capability enables direct data transmission between user memory spaces, avoiding redundant data movement, reducing memory bandwidth usage, and lowering memory capacity consumption.
• Shielding hardware differences and compatibility with multiple links to implement cross-device high-speed interconnection: The unidirectional communication library shields the underlying hardware differences of the Ascend series chips. You do not need to adapt the code to different chip architectures. At the communication link layer, it natively supports high-speed interconnection protocols such as RDMA and HCCS, with communication bandwidth up to 119 GB/s. It enables seamless high-speed interconnection across heterogeneous devices, meeting low-latency and high-throughput requirements.
• Simplified API design for in-depth adaptation to the open source ecosystem: The unilateral communication library uses a simplified API design, reducing the number of APIs to more than 10 core calls and providing comprehensive C++/Python language APIs. Currently, ModelArts has been deeply integrated with open source frameworks such as Mooncake and DeepLink. Mainstream inference engines such as vLLM and SGLang can directly call APIs of the unilateral communication library to implement efficient cross-device transmission of KV cache, reducing the memory access latency during foundation model inference by 20% and improving inference throughput.

Software Architecture

For details about the unilateral communication library, see #EN-US_TOPIC_0000002568092219/en-us_topic_0000002374418350_fig4440138542.

Core components:

• HIXL Engine: As the core transmission engine, it provides basic transmission APIs and supports multiple memory transmission types such as D2D, D2H, and H2D. It is compatible with various transmission protocols including HCCS and RDMA, enabling high-speed and reliable data transmission. It natively supports multiple data link types and is suitable for complex scenarios in homogeneous and heterogeneous clusters. It can efficiently adapt to link changes and schedule resources under dynamic cluster scaling, forming a reliable communication foundation for overall cluster operation.
• LLM-DataDist: It is built based on HIXL Engine and provides a set of data transmission interfaces that carry KV cache semantics. enabling fast and flexible integration with inference engines such as vLLM and SGLang.

Get Started

In the H2RD scenario (data is transmitted from the host to the remote device), the client initiates a data read/write request to the server.

1. Bind the Ascend device used by the process on the client and server, initialize the HIXL object, apply for memory, and register with the HIXL.
   • On the client side:

     // Bind the Ascend device used by the process.
     CHECK_ACL(aclrtSetDevice(device));
     
     // Initialize the HIXL engine with local_engine as the identifier. (Generally, the client side is a character string without a port IP address.)
     Hixl hixl_engine;
     std::map<AscendString, AscendString> options;
     CHECK_HIXL(hixl_engine.Initialize(local_engine, options));
     
     // Allocate the transfer buffer on the host for the local end of MEM_HOST registration and unilateral read/write.
     int32_t *host_buf = nullptr;
     CHECK_ACL(aclrtMallocHost(reinterpret_cast<void **>(&host_buf), sizeof(int32_t)));
     
     // Register the host address with HIXL so that the peer end can perform unidirectional access through the established link.
     MemDesc reg_desc{};
     reg_desc.addr = reinterpret_cast<uintptr_t>(host_buf);
     reg_desc.len = sizeof(int32_t);
     MemHandle handle = nullptr;
     CHECK_HIXL(hixl_engine.RegisterMem(reg_desc, MEM_HOST, handle));

   • On the server side:

     // Bind the Ascend device used by the process.
     CHECK_ACL(aclrtSetDevice(device));
     
     // Initialize the HIXL on the local_engine with a port as the connected end.
     Hixl hixl_engine;
     std::map<AscendString, AscendString> options;
     CHECK_HIXL(hixl_engine.Initialize(local_engine, options));
     
     // Allocate memory on the device and write the initial value 1 through H2D for client READ verification.
     constexpr int32_t kInit = 1;
     int32_t *dev_buf = nullptr;
     CHECK_ACL(aclrtMalloc(reinterpret_cast<void **>(&dev_buf), sizeof(int32_t), ACL_MEM_MALLOC_HUGE_ONLY));
     CHECK_ACL(aclrtMemcpy(dev_buf, sizeof(int32_t), &kInit, sizeof(int32_t), ACL_MEMCPY_HOST_TO_DEVICE));
     
     // Register the device address with HIXL so that the client can read and write the address range unilaterally.
     MemDesc reg_desc{};
     reg_desc.addr = reinterpret_cast<uintptr_t>(dev_buf);
     reg_desc.len = sizeof(int32_t);
     MemHandle handle = nullptr;
     CHECK_HIXL(hixl_engine.RegisterMem(reg_desc, MEM_DEVICE, handle));

2. The client initiates a link establishment request to the server.

   // Set up a transmission link with remote_engine (server listening address).
   CHECK_HIXL(hixl_engine.Connect(remote_engine));

3. Initiate a data read/write request from the client to the server. Check whether the read and write values are normal on the client and server.
   • On the client side:

     // Single-sided read: Read the pre-written 1 from the MEM_DEVICE of the peer server to the local host memory.
     TransferOpDesc xfer{reinterpret_cast<uintptr_t>(&host_buf), dev_buf, sizeof(int32_t)};
     CHECK_HIXL(hixl_engine.TransferSync(remote_engine, READ, {xfer}));
     printf("after READ: %d\n", *host_buf);
     
     // Single-side write: Write the preset host memory 2 to the MEM_DEVICE of the peer server.
     *host_buf = 2;
     CHECK_HIXL(hixl_engine.TransferSync(remote_engine, WRITE, {xfer}));

   • On the server side:

     // After the client completes the single-side write, the server reads the value from the device back to the host. After the client write is verified, the value is 2.
     int32_t host_val = 0;
     CHECK_ACL(aclrtMemcpy(&host_val, sizeof(int32_t), dev_buf, sizeof(int32_t), ACL_MEMCPY_DEVICE_TO_HOST));
     printf("after transfer: %d\n", host_val);

4. Release the link on the client.

   CHECK_HIXL(hixl_engine.Disconnect(remote_engine));

5. Deregister and release the memory on the client and server, destroy the HIXL engine, and unbind the HIXL engine from the device.
   • On the client side:

     CHECK_HIXL(hixl_engine.DeregisterMem(handle));
     CHECK_ACL(aclrtFreeHost(host_buf));
     hixl_engine.Finalize();
     CHECK_ACL(aclrtResetDevice(device));

   • On the server side:

     CHECK_HIXL(hixl_engine.DeregisterMem(handle));
     CHECK_ACL(aclrtFree(dev_buf));
     hixl_engine.Finalize();
     CHECK_ACL(aclrtResetDevice(device));