Class Description

InferenceRequest provides entries for initializing requests, setting tensors, and obtaining request information.

In the following sections, the P node refers to the prefill node, and the D node refers to the decode node.