Inference API

Description

Belongs to the service plane and uses the IP address and port number of the service plane.
It consists of a request and a response.

Request

After receiving an external request, the coordinator calculates the prefill and decode instances for serving the request based on the cluster status. Then, the request is sent to the corresponding prefill instance.

Standard inference (prefill-decode hybrid deployment) provides five types of inference APIs (TGI, vLLM, OpenAI, Triton, and MindIE native APIs). Therefore, in the prefill-decode disaggregation scenario, the coordinator needs to be compatible with these five types of APIs. After receiving an external request (RESTful request), the coordinator forwards the request to the prefill instance. The request URL and body are the same as those in the prefill-decode hybrid deployment scenario. However, some control information needs to be added to the HTTP header. For details about the control information, see Table 1.

**Table 1** Description of the HTTP header in a prefill request
Field	Mandatory/Optional	Description	Value Type
req-id	Mandatory	Globally unique request ID.	The value is a string of digits and must be globally unique.
d-target	Mandatory	IP address of the target decode instance.	IP address.
req-type	Mandatory	Request type.	The value can only be prefill.
is-recompute	Optional (mandatory in the recomputation scenario)	Whether to recompute.	true: recomputation false: non-recomputation

Request example:

curl -i -H "req-id: 12345" -H "req-type: prefill" -H "d-target: 172.17.0.9" -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '
{
    "model": "llama3_70b",
    "messages": [{
        "role": "system",
        "content": "You are a helpful assistant."
    }],
    "max_tokens": 20,
    "presence_penalty": 1.03,
    "frequency_penalty": 1.0,
    "seed": null,
    "temperature": 0.5,
    "top_p": 0.95,
    "stream": true
}' https://172.17.0.10:1025/v1/chat/completions

Response

The response to a request occurs in the prefill and decode instances, and the response content of the two instances is different.

Response in the prefill instance:

Response body:

{
    "reqId": "12345",
    "isStream": true,
    "output": "data:{...}",
    "isLastResp": false
}

For details about the fields in the response body, see Table 2.

**Table 2** Response fields of the prefill instance
Field	Mandatory/Optional	Description	Value Type
reqId	Mandatory	Response to RequestID.	Character string of the numeric type.
isStream	Mandatory	Whether the request is a streaming request.	Bool. true: streaming request. false: non-streaming request.
output	Optional	Content of the response to a request.	Character string.
isLastResp	Mandatory	Whether the current response is the last response.	Bool. true: The response is the last response. false: The response is not the latest response.

The coordinator only forwards external requests after ingesting the HTTP header and does not parse the content of these requests. Consequently, it cannot distinguish between streaming and non-streaming requests, as some are differentiated by their URLs and others by their bodies. Currently, the coordinator does not parse the request bodies. Additionally, the coordinator returns different information for streaming and non-streaming requests. In the response for the prefill instance, the coordinator directly notifies the prefill instance about whether the requests are streaming or non-streaming.

The coordinator responds to the external requester with the inference result, but the response content comes from the inference instance. In a prefill response, the response content is displayed in the output field. The coordinator does not need to parse the output content. Instead, it returns the content to the requester in an appropriate mode (streaming or non-streaming, first token, unique token, or different API types). Different modes may cause varying behaviors of the coordinator.

For a streaming request, the prefill instance outputs the content of the first token. Using Triton as an example, the output is as follows:

data:{"id":"a123","model_name":"llama3-70b","model_version":null,"text_output":"am","details":{"generated_tokens":1,"first_token_cost":null,"decode_cost":null,"batch_size":1,"queue_wait_time":10},"prefillTime":26.96,"decodeTime":null}

For a non-streaming request, the prefill instance does not return the first token. Generally, there is no output field.
However, there is a special case, that is, only one token is returned for the entire inference request. In this case, the request is terminated in the prefill instance. isLastResp is used to identify this special case. When isLastResp is true, the output of a non-streaming request displays the complete inference result.

Response in the decode instance:
There is a persistent connection between the coordinator and the decode instance. Each time the decode instance generates an inference result, the result is returned to the coordinator through the persistent connection.

Operation type: GET

URL: https://{ip}:{port}/dresult

Request example:
```
GET https://{ip}:{port}/dresult
```
The response is of the chunk type. The HTTP header contains the following two fields:
```
Content-Type: text/event-stream
Transfer-Encoding: chunked
```
The return result of a single request consists of two parts: size and data.

size is a hexadecimal character string (without 0x), indicating data length. data is the content that is actually transmitted. Both size and data end with \r\n.
```
18\r\n{"aaa":"bbb","ccc":"ddd"}\r\n
```
- 18 is the hexadecimal representation of 24.
- The preceding HTTP chunk principles have been encapsulated in the HTTP library. You do not need to pay attention to them in the development phase.

Parent topic: Prefill-Decode Disaggregation APIs