Querying Model Configuration Data

Function

Queries model configuration data.

This API is scheduled for deprecation.

Format

Operation type: GET

URL: https://{ip}:{port}/v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/config

Replace {ip} and {port} with the IP address and port number of the service plane, that is, ipAddress and port.
The ${MODEL_NAME} field specifies the name of the model to be queried.
The [/versions/${MODEL_VERSION}] field is not supported currently and is not passed.

Request Parameters

None

Usage Example

Request example:

GET https://{ip}:{port}/v2/models/llama3-70b/config

Response example:

{
    "model_name": "llama3-70b",
    "input_datatype": "INT64",
    "output_datatype": "INT64",
    "max_seq_len": 2560,
    "npu_mem_size": 8,
    "cpu_mem_size": 5,
    "world_size": 8,
    "model_weight_path": "llama3-70b-safetensors",
    "model_instance_type": "Standard"
}

Response status code: 200

Output Description

Parameter	Type	Description
model_name	String	Name of the model selected for inference.
input_datatype	String	Input data type.
output_datatype	String	Output data type.
max_seq_len	Integer	Maximum sequence length.
npu_mem_size	Integer	Maximum size of the KV cache that can be allocated on a single NPU, in GB.
cpu_mem_size	Integer	Maximum size of the KV cache that can be allocated on a CPU, in GB.
world_size	Integer	Number of cards used for inference.
model_weight_path	String	Name of the last-level directory in the model weight file path.
model_instance_type	String	Model type.

Parent topic: APIs Compatible with Triton