Development Workflow

Introduction

To simplify the description, inference frameworks are broken down into the following modules:

• Resource initialization module
• KV cache management module for creating, allocating (in the PagedAttention scenario), and destroying KV cache memory
• Model inference module
• Resource release module

This section guides you through enabling LLM-DataDist capabilities within an inference framework.

Development Procedure

1. Locate the resource initialization module in your inference framework. Within this phase, call the LLM-DataDist initialization and link establishment APIs.
2. Locate the KV cache management module. Within this phase, call the LLM-DataDist KV cache allocation API. After allocation, convert the obtained KV cache into the type required by your specific inference framework.
3. The inference framework must separate the Prefill and Decode phases. Deploy the inference scripts for these phases on different cluster nodes. Before the Decode phase begins, it must receive the output from the Prefill phase as its input. At this point, use the LLM-DataDist KV cache transmission API to pull or push the KV cache from or to the Prefill side.
4. Execute the inference scripts for Prefill and Decode separately.
5. Release LLM-DataDist-related resources within the framework's resource release module.