Overview
The unilateral communication library provides simple, reliable, and efficient point-to-point data transfer for cluster scenarios through simple APIs, bridging multiple AI applications and transmission links. It can be used in various scenarios such as large language model (LLM) prefill-decode (PD) disaggregation, reinforcement learning (RL) post-training parameter switching, and model parameter caching.
This document is a development guide for the unilateral communication library, showing developers how to use its APIs for inter-cluster data transmission and to build a disaggregated LLM inference framework.
Manual |
Description |
|---|---|
Unilateral Communication Library (C++) |
Describes Huawei Xfer Library (HIXL) APIs for C++, including link management, memory management, and data transmission. In distributed memory pool scenarios, HIXL provides a pure transmission capability based on local and remote addresses. D2D, D2H, and H2D transmissions are supported. Describes how to perform link management and key-value (KV) cache management via the C++-based LLM-DataDist APIs. This scenario supports unilateral link establishment. Decode and Prompt can bidirectionally pull and push the KV cache. |
Unilateral Communication Library (Python) |
Describes how to perform link management and KV cache management via the Python-based LLM-DataDist APIs in CacheManager mode. Both unilateral and bilateral link establishments are supported. That is, all LLM-DataDist instances can initiate link establishments simultaneously. Decode and Prompt can bidirectionally pull and push the KV cache. D2D, D2H, and H2D transmissions are supported. |