API for Querying Service Management and Control Metrics (Prometheus Format)

To use this API, ensure that the service management and control function is enabled before starting a service. To enable the service management and control function, run the following command:

export MIES_SERVICE_MONITOR_MODE=1

Function

Queries management and control metrics of inference services.

Format

Operation type: GET

URL: https://{ip}:{port}/metrics

The {ip} field prioritizes reading the environment variable MIES_CONTAINER_MANAGEMENT_IP. If the environment variable does not exist, managementIpAddress in the configuration file is used. If managementIpAddress does not exist in the configuration file, ipAddress in the configuration file is used.
The {port} field reads the metricsPort parameter in the configuration file.

Request Parameters

None

Usage Example

Request example:

GET https://{ip}:{port}/metrics

Response example:

# HELP request_received_total Number of requests received so far.
# TYPE request_received_total counter
request_received_total{model_name="llama3-70b"} 3188
# HELP request_success_total Number of requests proceed successfully so far.
# TYPE request_success_total counter
request_success_total{model_name="llama3-70b"} 2267
# HELP request_failed_total Number of requests failed so far.
# TYPE request_failed_total counter
request_failed_total{model_name="llama3-70b"} 0
# HELP num_preemptions_total Cumulative number of preemption from the engine.
# TYPE num_preemptions_total counter
num_preemptions_total{model_name="llama3-70b"} 637
# HELP num_requests_running Number of requests currently running on NPU.
# TYPE num_requests_running gauge
num_requests_running{model_name="llama3-70b"} 0
# HELP num_requests_waiting Number of requests waiting to be processed.
# TYPE num_requests_waiting gauge
num_requests_waiting{model_name="llama3-70b"} 0
# HELP num_requests_swapped Number of requests swapped to CPU.
# TYPE num_requests_swapped gauge
num_requests_swapped{model_name="llama3-70b"} 0
# HELP prompt_tokens_total Number of prefill tokens processed.
# TYPE prompt_tokens_total counter
prompt_tokens_total{model_name="llama3-70b"} 9564
# HELP generation_tokens_total Number of generation tokens processed.
# TYPE generation_tokens_total counter
generation_tokens_total{model_name="llama3-70b"} 84425
# HELP avg_prompt_throughput_toks_per_s Average prefill throughput in tokens/s.
# TYPE avg_prompt_throughput_toks_per_s gauge
avg_prompt_throughput_toks_per_s{model_name="llama3-70b"} 0.586739718914032
# HELP avg_generation_throughput_toks_per_s Average generation throughput in tokens/s.
# TYPE avg_generation_throughput_toks_per_s gauge
avg_generation_throughput_toks_per_s{model_name="llama3-70b"} 2.375296831130981
# HELP failed_request_perc Requests failure rate. 1 means 100 percent usage.
# TYPE failed_request_perc gauge
failed_request_perc{model_name="llama3-70b"} 0
# HELP npu_cache_usage_perc NPU KV-cache usage. 1 means 100 percent usage.
# TYPE npu_cache_usage_perc gauge
npu_cache_usage_perc{model_name="llama3-70b"} 1
# HELP cpu_cache_usage_perc CPU KV-cache usage. 1 means 100 percent usage.
# TYPE cpu_cache_usage_perc gauge
cpu_cache_usage_perc{model_name="llama3-70b"} 0
# HELP npu_prefix_cache_hit_rate NPU prefix cache block hit rate.
# TYPE npu_prefix_cache_hit_rate gauge
npu_prefix_cache_hit_rate{model_name="llama3-70b"} 0.5
# HELP time_to_first_token_seconds Histogram of time to first token in seconds.
# TYPE time_to_first_token_seconds histogram
time_to_first_token_seconds_count{model_name="llama3-70b"} 2523
time_to_first_token_seconds_sum{model_name="llama3-70b"} 9740.00200343132
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.001"} 0
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.005"} 0
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.01"} 0
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.02"} 0
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.04"} 0
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.06"} 10
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.08"} 54
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.1"} 104
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.25"} 256
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.5"} 256
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="0.75"} 276
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="1"} 321
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="2.5"} 628
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="5"} 1148
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="7.5"} 2523
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="10"} 2523
time_to_first_token_seconds_bucket{model_name="llama3-70b",le="+Inf"} 2523
# HELP time_per_output_token_seconds Histogram of time per output token in seconds.
# TYPE time_per_output_token_seconds histogram
time_per_output_token_seconds_count{model_name="llama3-70b"} 85800
time_per_output_token_seconds_sum{model_name="llama3-70b"} 4445.857012826018
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.01"} 0
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.025"} 0
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.05"} 3
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.075"} 12
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.1"} 40283
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.15"} 83145
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.2"} 83339
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.3"} 83339
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.4"} 83539
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.5"} 85139
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="0.75"} 85740
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="1"} 85800
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="2.5"} 85800
time_per_output_token_seconds_bucket{model_name="llama3-70b",le="+Inf"} 85800
# HELP e2e_request_latency_seconds Histogram of end to end request latency in seconds.
# TYPE e2e_request_latency_seconds histogram
e2e_request_latency_seconds_count{model_name="llama3-70b"} 2267
e2e_request_latency_seconds_sum{model_name="llama3-70b"} 12684.5319980979
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="1"} 27
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="2.5"} 268
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="5"} 712
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="10"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="15"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="20"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="30"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="40"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="50"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="60"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-70b",le="+Inf"} 2267
# HELP request_prompt_tokens Number of prefill tokens processed.
# TYPE request_prompt_tokens histogram
request_prompt_tokens_count{model_name="llama3-70b"} 3188
request_prompt_tokens_sum{model_name="llama3-70b"} 9564
request_prompt_tokens_bucket{model_name="llama3-70b",le="10"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="50"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="100"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="200"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="500"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="1000"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="2000"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="5000"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="10000"} 3188
request_prompt_tokens_bucket{model_name="llama3-70b",le="+Inf"} 3188
# HELP request_generation_tokens Number of generation tokens processed.
# TYPE request_generation_tokens histogram
request_generation_tokens_count{model_name="llama3-70b"} 2267
request_generation_tokens_sum{model_name="llama3-70b"} 84425
request_generation_tokens_bucket{model_name="llama3-70b",le="10"} 0
request_generation_tokens_bucket{model_name="llama3-70b",le="50"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="100"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="200"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="500"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="1000"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="2000"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="5000"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="10000"} 2267
request_generation_tokens_bucket{model_name="llama3-70b",le="+Inf"} 2267

Response status code: 200

Output Description

Return Value	Type	Description
request_received_total	Counter	Number of inference requests received by the server. model_name: user-defined model name, which is used in all example responses.
request_success_total	Counter	Number of successful inference requests on the server.
request_failed_total	Counter	Number of failed inference requests on the server.
num_requests_running	Gauge	Number of running requests on the server.
num_requests_waiting	Gauge	Number of waiting requests on the server.
num_requests_swapped	Gauge	Number of requests that are currently swapped to the CPU by the server.
num_preemptions_total	Counter	Number of preemption requests triggered by the server.
prompt_tokens_total	Counter	Total number of prompt tokens for all requests processed by the server.
generation_tokens_total	Counter	Total number of tokens generated by the server after inference.
avg_prompt_throughput_toks_per_s	Gauge	Latest average prefill throughput. The unit is tokens/s. This metric reflects the batch execution throughput collected by the system and does not account for the impact of framework features such as cross-machine communication or asynchronous dual delivery. It is for reference only. For example, when asynchronous dual delivery is enabled, its effects are not included in the calculation, which results in a distorted metric in this scenario.
avg_generation_throughput_toks_per_s	Gauge	Latest average token throughput. The unit is tokens/s. This metric reflects the batch execution throughput collected by the system and does not account for the impact of framework features such as cross-machine communication or asynchronous dual delivery. It is for reference only. For example, when asynchronous dual delivery is enabled, its effects are not included in the calculation, which results in a distorted metric in this scenario.
failed_request_perc	Gauge	Failure rate of inference requests on the server. Proportion of failed inference requests = Number of failed requests/Number of received requests. The value 1 indicates 100%.
npu_cache_usage_perc	Gauge	NPU usage of the current KV cache. The value 1 indicates 100%.
cpu_cache_usage_perc	Gauge	CPU memory usage of the current KV cache. The value 1 indicates 100%.
npu_prefix_cache_hit_rate	Gauge	Hit rate of the prefix cache on the NPU. The value 1 indicates 100%.
time_to_first_token_seconds	Histogram	Time to first token (TTFT), indicating the time consumed for generating the first token for inference. The unit is second. The TTFT is the pure inference time, excluding the time for HTTP communication, tokenizer, and KV cache transmission in prefill-decode disaggregation. time_to_first_token_seconds_count: number of requests that have measured TTFT so far. time_to_first_token_seconds_sum: sum of TTFT of all requests so far. time_to_first_token_seconds_bucket: TTFT of requests measured by histogram buckets so far. le: less than or equal to the boundary of a histogram bucket. NOTE: All the le parameters in the response example have the same meaning.
time_per_output_token_seconds	Histogram	Time Per Output Token (TPOT), which indicates the interval for generating two consecutive tokens, in seconds. The TPOT is the pure inference time, excluding the time for HTTP communication and KV cache transmission in prefill-decode disaggregation. time_per_output_token_seconds_count: number of tokens that have measured TPOT so far. time_per_output_token_seconds_sum: sum of TPOT of all tokens so far. time_per_output_token_seconds_bucket: TPOT measured by histogram buckets so far.
e2e_request_latency_seconds	Histogram	End-to-end latency, indicating how long it takes from receiving a request to executing it, in seconds. e2e_request_latency_seconds_count: number of requests that have measured end-to-end latency so far. e2e_request_latency_seconds_sum: sum of end-to-end latency of all requests so far. e2e_request_latency_seconds_bucket: end-to-end latency of requests measured by histogram buckets so far.
request_prompt_tokens	Histogram	Number of tokens input by a request, that is, number of tokens generated after the input prompt is processed by the tokenizer. request_prompt_tokens_count: number of requests that have queried current metrics so far. request_prompt_tokens_sum: sum of input tokens of all requests that have queried current metrics so far. request_prompt_tokens_bucket: number of input tokens measured by histogram buckets so far.
request_generation_tokens	Histogram	Number of tokens output by a request, that is, number of tokens generated after model inference. request_generation_tokens_count: number of requests that have queried current metrics so far. request_generation_tokens_sum: sum of output tokens of all requests that have queried current metrics so far. request_generation_tokens_bucket: number of output tokens measured by histogram buckets so far.

Parent topic: Internal Endpoint APIs