TGI EndPoint information query
- This API is scheduled for deprecation.
- The API is compatible with the return format of the TGI API to the maximum extent. For the return fields that are not supported by Server, null is returned.
Function
Queries the TGI Endpoint information.
Format
Operation type: GET
URL: https://{ip}:{port}/info
- The {ip} field preferentially reads the environment variable MIES_CONTAINER_MANAGEMENT_IP. If the environment variable does not exist, managementIpAddress in the configuration file is used. If managementIpAddress does not exist in the configuration file, ipAddress in the configuration file is used.
- The {port} field preferentially reads the managementPort parameter in the configuration file. If the configuration file does not contain the managementPort parameter, the port parameter in the configuration file is used.
Request Parameters
None
Usage Example
Request example:
GET https://{ip}:{port}/info
Response example:
{
"docker_label": null,
"max_batch_total_tokens": 8192,
"max_best_of": 1,
"max_concurrent_requests": 200,
"max_stop_sequences": null,
"max_waiting_tokens": null,
"sha": null,
"validation_workers": null,
"version": "1.0.0",
"waiting_served_ratio": null,
"models": [
{
"model_device_type": "npu",
"model_dtype": "float16",
"model_id": "deepseek",
"model_pipeline_tag": "text-generation",
"model_sha": null,
"max_total_tokens": 2560
}
],
"max_input_length": 2048
}
Response status code: 200
Output Description
Parameter |
Type |
Description |
|
|---|---|---|---|
docker_label |
String |
Not supported. By default, null is returned. |
|
max_batch_total_tokens |
Integer |
Equals the value of maxPrefillTokens. |
|
max_best_of |
Integer |
Not supported. By default, 1 is returned, that is, only one inference result is returned each time. |
|
max_concurrent_requests |
Integer |
Maximum number of concurrent requests. The value is the same as that of maxBatchSize. |
|
max_stop_sequences |
Integer |
Not supported. By default, null is returned. |
|
max_waiting_tokens |
Integer |
Not supported. By default, null is returned. |
|
sha |
String |
Not supported. By default, null is returned. |
|
validation_workers |
Integer |
Not supported. By default, null is returned. |
|
version |
String |
Version number. |
|
waiting_served_ratio |
Float |
Not supported. By default, null is returned. |
|
models |
List |
Model configuration. |
|
- |
model_device_type |
String |
Type of the device where the model runs. The default value is npu. |
model_dtype |
String |
Model data type. The value is torch_dtype in the config.json file. |
|
model_id |
String |
Model name. |
|
model_pipeline_tag |
String |
Model task type. The default value is text-generation. |
|
model_sha |
String |
Not supported. By default, null is returned. |
|
max_total_tokens |
Integer |
Maximum number of inference tokens. The value is the same as maxSeqLen. |
|
max_input_length |
Integer |
Maximum input length. The value is the same as maxInputTokenLen. |
|