Development Workflow

Introduction

To simplify the description, inference frameworks are broken down into the following modules:

• Resource initialization module
• KV cache management module for creating, allocating (in the PagedAttention scenario), and destroying KV cache memory
• Model inference module
• Resource release module

This section guides you through enabling LLM-DataDist capabilities within an inference framework.

Development Procedure

1. Locate the resource initialization module in your inference framework. Within this phase, call the LLM-DataDist initialization API.
2. Locate the KV cache management module in your inference framework. Within this phase, call the LLM-DataDist registration API to register the allocated memory with LLM-DataDist.
3. Separate the Prefill and Decode phases for the inference framework, deploy their inference scripts on different cluster nodes, and establish links between them as required.
4. Before the Decode phase begins, receive the output from the Prefill phase as its input. At this point, use the LLM-DataDist KV cache transmission API to pull or push the KV cache from or to the Prefill side.
5. Execute the inference scripts for Prefill and Decode separately.
6. Release LLM-DataDist-related resources within the framework's resource release module.