Sample Reference

This sample uses the Llama model from Transformers to highlight the script changes before and after PD disaggregation, providing guidance on adapting non-disaggregated scripts to PD-disaggregated ones. (Click Gitee, download the sample package of the corresponding version based on the version mapping table, and obtain the sample from the npu_tuned_model/llm/llama/benchmark/pd_separate directory.) In this sample, the full and incremental models are deployed on separate cluster nodes for execution.

The following steps illustrate how the disaggregated script is scheduled by the service layer during the inference process:

1. When a user request arrives, the service layer schedules it to the full cluster, where the full script executes inference. The information required by the incremental script is transmitted to the incremental script's execution node. Meanwhile, the full cluster nodes can accept new user requests issued by the service layer.
2. The incremental cluster receives the corresponding request information from the full cluster, pulls the precomputed KV cache for the request (generated on the full cluster nodes), batches requests according to the incremental model's batch size, and performs incremental inference.
3. When a request completes inference on the incremental cluster and a batch slot becomes available, the incremental cluster accepts new requests from the full cluster and repeats steps 2 and 3.
4. The full cluster repeats step 1, and the incremental cluster repeats steps 2 and 3 until the service terminates and both clusters exit.

Click Gitee, download the sample package of the matching version based on the tag name, and obtain the sample from the python/level1_single_api/10_llm_data_dist directory.