Common Prefixes

Application Scenarios

A common prefix refers to the identical initial segment shared by multiple input prompts during model inference.

The KV cache generated from the common prefix can be copied to the KV cache of a new user request for inference.

Related APIs

Example

1
2
3
4
5

ret = llmDataDist.CopyKvCache(src_cache, dst_cache, 0, 0);
if (ret != LLM_SUCCESS) {
    printf("[ERROR] CopyKvCache failed, ret = %u\n", ret);
    return -1;
}

Troubleshooting

For details about troubleshooting, see Error Codes.