LLM-DataDist API List

LLM-DataDist is a LLM distributed cluster and data acceleration component. It provides clustered KV data management capabilities and supports disaggregated deployment of full graphs and incremental graphs.

The following products are supported:
- Atlas A2 training products/Atlas A2 inference products: For the Atlas A2 training products/Atlas A2 inference products, only the Atlas 800I A2 inference server, Atlas 300I A2 inference card, and A200I A2 Box heterogeneous components are supported. When the HCCS transmission protocol is used in the server, only D2D transmission is supported.
- Atlas A3 training products/Atlas A3 inference products: When the HCCS transmission protocol is used, the host memory cannot be used as the remote cache.

Currently, Python 3.9, Python 3.11, and Python 3.12 are supported. For details about how to install it, visit the Python official website: https://www.python.org/.
A maximum of 50 GB device memory and 20 GB host memory can be registered. Larger registered memory results in higher OS memory usage.

The LLM-DataDist APIs are listed as follows.

LLM-DataDist

**Table 1** LLM-DataDist APIs
API	Description
LLM-DataDist Constructor	Constructs an LLM-DataDist instance.
init	Initializes an LLM-DataDist instance.
finalize	Releases an LLM-DataDist instance.
link_clusters	Establishes a link unilaterally. The request is initiated by the client. The server/client roles are independent of the prompt/decoder roles. Setting listen_ip_info on a node designates the node as the server (port listens for links).
unlink_clusters	Disconnects a link unilaterally. The request can be initiated by the client to notify the server of disconnection. Alternatively, both the client and server can initiate forcible disconnection to clean up only local links.
switch_role	Switches the role of the current LLM-DataDist instance. You can also configure switch_options to switch the role to client or server.
link	Establishes a link by establishing a communicator in bilateral link establishment scenarios.
unlink	Disconnects a link in bilateral link establishment scenarios.
query_register_mem_status	Queries the status of the registered memory in bilateral link establishment scenarios.
cache_manager	Obtains the CacheManager instance.

LLMConfig

**Table 2** LLMConfig APIs
API	Description
LLMConfig Constructor	Constructs an LLMConfig instance.
generate_options	Generates a configuration item dictionary.
device_id	Sets the device ID of the current process, corresponding to the underlying ge.exec.deviceId configuration item.
sync_kv_timeout	Sets the timeout interval for APIs (such as the KV pull API), corresponding to the underlying llm.SyncKvCacheWaitTime configuration item.
ge_options	Configures additional GE configuration items.
listen_ip_info	Sets the cluster listening information on the PROMPT side, corresponding to the underlying llm.listenIpInfo configuration item.
enable_cache_manager	Specifies whether to enable the CacheManager mode, corresponding to the underlying llm.EnableCacheManager configuration item. Set this item to True.
mem_pool_cfg	Configures the memory pool when CacheManager is enabled.
host_mem_pool_cfg	Configures the host memory pool when CacheManager is enabled.
enable_remote_cache_accessible	Specifies whether to enable the function of directly accessing the remote cache when CacheManager is enabled.
rdma_traffic_class	Sets the traffic class of the RDMA NIC when CacheManager is enabled.
rdma_service_level	Sets the service level of the RDMA NIC when CacheManager is enabled.
local_comm_res	Configures local communication resources when CacheManager is enabled.
link_total_time	Sets the total timeout interval for HCCL link establishment failures.
link_retry_count	Sets the number of retries upon HCCL link establishment failures.

CacheManager

**Table 3** CacheManager APIs
API	Description
CacheManager Constructor	Constructs a CacheManager instance.
allocate_cache	Allocates a cache. After successful allocation, the cache is simultaneously referenced by cache_id and cache_keys. The resources occupied by the cache are not actually released until all these references are removed.
deallocate_cache	Deallocates a cache. If the cache is associated with a cache key during allocation, the actual release is delayed until all cache keys are pulled or remove_cache_key is executed.
remove_cache_key	Removes a cache key. After a cache key is removed, the corresponding cache can no longer be pulled by pull_cache.
pull_cache	Pulls the cache from the peer node to the local cache based on the cache key.
copy_cache	Copies the cache.
allocate_blocks_cache	Allocates the cache of multiple blocks in the PagedAttention scenario. After the cache is successfully allocated, you can call deallocate_blocks_cache to release the memory.
deallocate_blocks_cache	Releases the cache allocated by allocate_blocks_cache in the PagedAttention scenario.
pull_blocks	Pulls the cache from the peer node to the local cache via a block list based on BlocksCacheKey in the PagedAttention scenario.
copy_blocks	Copies blocks in the PagedAttention scenario.
swap_blocks	Swaps in and out the cpu_cache and npu_cache.
register_cache	Registers a self-allocated memory.
register_blocks_cache	Registers a self-allocated memory in the PagedAttention scenario.
transfer_cache_async	Asynchronously transmits the KV cache in a hierarchical manner.
push_blocks	Pushes the cache from the local node to the remote cache via a block list based on BlocksCacheKey in the PagedAttention scenario.
push_cache	Pushes the cache from the local node to the remote cache based on the cache key (CacheKey).
remap_registered_memory	During LLM inference, if a memory uncorrectable error (UCE) occurs (that is, ACL_ERROR_RT_DEVICE_MEM_ERROR is returned), the upper-layer framework needs to first determine whether the faulty memory is a KV cache memory. If it is not a KV cache memory, refer to the torch_npu.npu.restart_device API description in Ascend Extension for PyTorch Custom API Reference to obtain and fix the erroneous virtual address of the memory UCE. If it is a KV cache memory, additionally call this API to restore the KV cache memory registered with the NIC. This API is reserved, and the function is not supported currently.
unregister_cache	Deregisters a self-allocated memory.

Cache

**Table 4** Cache APIs
API	Description
Cache Constructor	Constructs a Cache instance.
cache_id	Obtains the cache ID.
cache_desc	Obtains the cache description.
tensor_addrs	Obtains the cache address.
create_cpu_cache	Creates a CPU cache.

LLMClusterInfo

**Table 5** LLMClusterInfo APIs
API	Description
LLMClusterInfo Constructor	Constructs an LLMClusterInfo instance.
remote_cluster_id	Sets the peer cluster ID.
append_local_ip_info	Adds the IP address of the local cluster.
append_remote_ip_info	Adds the IP address of the remote cluster.

CacheTask

**Table 6** CacheTask APIs
API	Description
CacheTask Constructor	Constructs a CacheTask instance.
synchronize	Waits until the transfer at all layers is complete and obtains the overall execution result.
get_results	Waits until the transfer at all layers is complete and obtains the execution result of each TransferConfig.

Other APIs

**Table 7** Other APIs
API	Description
LLMRole	Enumerates values of LLMRole.
RegisterMemStatus	Enumerates values of RegisterMemStatus.
Placement	Represents the device type where the cache is located. It is a field in CacheDesc.
CacheDesc	Constructs a CacheDesc instance.
MemType	Indicates the location of the address. It is usually used as the type of the mem_type parameter of MemInfo.
MemInfo	Constructs a MemInfo instance, which is typically used as a parameter type in the remap_registered_memory API of CacheManager.
CacheKey	Constructs a CacheKey instance.
CacheKeyByIdAndIndex	Constructs a CacheKeyByIdAndIndex instance, which is usually used as the parameter type in pull_cache and push_cache of CacheManager.
BlocksCacheKey	Constructs a BlocksCacheKey instance in the PagedAttention scenario.
LayerSynchronizer	Waits for the specified layer of the model to complete execution. Users need to inherit from LayerSynchronizer and implement this API. This API is called during the execution of transfer_cache_async. When the API returns a success status, the cache transmission for the current layer begins.
TransferConfig	Construct a TransferConfig instance.
TransferWithCacheKeyConfig	Constructs a TransferWithCacheKeyConfig instance.
LLMException	Obtains the error code of the exception. For details about the error code list, see LLMStatusCode.
LLMStatusCode	Enumerates values of LLMStatusCode.
DataType	Enumerates values of DataType.

Parent topic: HIXL API Reference (Python)