Early Termination of a Request
Currently, this API supports only early termination of Triton inference requests.
Function
Terminates a request in advance based on the Triton API definition.
Format
Operation type: POST
URL: https://{ip}:{port}/v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/stopInfer
- Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.
- The ${MODEL_NAME} field specifies the name of the model to be queried.
- The [/versions/${MODEL_VERSION}] field is not supported currently and is not passed.
Request Parameters
Parameter |
Mandatory/Optional |
Description |
Value |
|---|---|---|---|
id |
Mandatory |
Inference request ID |
The value is a string of a maximum of 256 characters. |
Usage Example
Request example:
POST https://{ip}:{port}/v2/models/llama3-70b/stopInfer
Request body:
{
"id":"a123"
}
Response example:
{
"id":"a123"
}
Response status code: 200
Output Description
Return Value |
Type |
Description |
|---|---|---|
id |
String |
ID of the inference request that is successfully terminated. In 2.1.RC1 and earlier versions, after this API is called, the inference is not terminated immediately. Instead, the inference is processed together with scheduling in the executor, and an invalid token is returned. In 2.2.RC1 and later versions, the inference is terminated immediately after this API is called, and no invalid token is returned. |