Reading Guide

If you are not familiar with LLM-DataDist or are reading this document for the first time, you are advised to read Table 1 first to quickly learn the manual content and access the sections as needed.

Table 1 Reading guide

Scenario

Section

Description

Understanding LLM-DataDist

Introduction

This section describes the basic concepts, functions, and advantages of LLM-DataDist, and introduces its application scenarios.

Building a disaggregated LLM inference framework based on LLM-DataDist

Quick Start

This section describes the overall development process and environment preparation for building a disaggregated LLM inference framework using LLM-DataDist, and provides complete executable reference samples.

Learning about LLM-DataDist API capabilities

Functions

This section describes the functions of LLM-DataDist APIs and how to implement them.

Exploring advanced features of LLM-DataDist

Special Topics

This section describes advanced features for building a disaggregated LLM inference framework based on LLM-DataDist, including KV cache swapping and common prefixes.

Acquiring extended knowledge

Appendices

This section describes extended knowledge mentioned in this document, including basic concepts and LLM inference process.

Troubleshooting

LLMStatusCode

This section describes API error codes, along with cause analysis and solutions.