API for Collecting Static Configurations

Function

Collects static configurations.

Format

Operation type: GET

URL: https://{ip}:{port}/v1/config

The {ip} field prioritizes reading the environment variable MIES_CONTAINER_MANAGEMENT_IP. If the environment variable does not exist, managementIpAddress in the configuration file is used. If managementIpAddress does not exist in the configuration file, ipAddress in the configuration file is used.
The {port} field prioritizes reading the managementPort parameter in the configuration file. If the configuration file does not contain the managementPort parameter, the port parameter in the configuration file is used.

Request Parameters

None

Usage Example

Request example:

GET https://{ip}:{port}/v1/config

Response example:

{
    "modelName": "llama3-70b",
    "maxSeqLen": 2560,
    "npuMemSize": 8,
    "cpuMemSize": 5,
    "worldSize": 8,
    "maxOutputLen": 512,
    "cacheBlockSize": 128
}

Output Description

Field	Type	Description
modelName	String	Name of the model selected for inference.
maxSeqLen	uint32_t	Maximum sequence length.
npuMemSize	uint32_t	Maximum size of the KV cache that can be allocated on an NPU.
cpuMemSize	uint32_t	Maximum size of the KV cache that can be allocated on a CPU.
worldSize	uint32_t	Number of cards used for inference.
maxOutputLen	uint32_t	Maximum output length.
cacheBlockSize	uint32_t	Size of a KV cache block.

Parent topic: Prefill-Decode Disaggregation APIs