Development Workflow

Overview

To simplify the description, inference frameworks are broken down into the following modules:

• Resource initialization module
• KV cache management module for creating, allocating (in PagedAttention scenarios), and destroying KV cache memory
• Model inference module
• Resource release module

This section guides you through enabling LLM-DataDist capabilities within an inference framework.

Development Procedure

1. Locate the resource initialization module in your inference framework. In this stage, call the LLM-DataDist initialization and link establishment APIs.
2. Locate the KV cache management module. In this stage, call the LLM-DataDist KV cache allocation API. After allocation, convert the obtained KV cache into the type required by your specific inference framework.
3. The inference framework must disaggregate the Prefill and Decode stages. Deploy the inference scripts for these stages on different cluster nodes. Before the Decode stage begins, it must receive the output from the Prefill stage as its input. At this point, use the LLM-DataDist KV cache transmission API to pull or push the KV cache from or to the Prefill stage.
4. Execute the inference scripts for Prefill and Decode separately.
5. Release LLM-DataDist resources in the resource release module.