Slot Statistics API

Queries slot statistics based on the Triton format.

This API is scheduled for deprecation.

Operation type: GET

URL: https://{ip}:{port}/v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/getSlotCount

The {ip} field preferentially reads the environment variable MIES_CONTAINER_MANAGEMENT_IP. If the environment variable does not exist, managementIpAddress in the configuration file is used. If managementIpAddress does not exist in the configuration file, ipAddress in the configuration file is used.
The {port} field preferentially reads the managementPort parameter in the configuration file. If the configuration file does not contain the managementPort parameter, the port parameter in the configuration file is used.
The ${MODEL_NAME} field specifies the name of the model to be queried.
The [/versions/${MODEL_VERSION}] field is not supported currently and is not passed.

None

Request example:

GET https://{ip}:{port}/v2/models/llama3-70b/getSlotCount

Response example:

{
    "total_slots": 50,
    "free_slots": 50,
    "available_tokens_length": 30720
}

Response status code: 200

Return Value	Type	Description
total_slots	Integer	Maximum batch size supported by the inference service. The value is the same as that of the maxBatchSize field in the configuration file.
free_slots	Integer	Remaining slots. The value is obtained from the parameters managed by the scheduling module. The value of total_slots minus the value of free_slots is the number of requests that are being processed by the scheduler. The value is an instantaneous value.
available_tokens_length	Integer	Number of remaining tokens that can be used in the KV cache.

Parent topic: Native MindIE APIs