API for Querying Management and Control Metrics

Function

Queries service management and control metrics of a cluster. The return is in Prometheus metrics format.

To use this API, ensure that the service management and control function is enabled before starting a service. The following is the command to enable service management and control:

export MIES_SERVICE_MONITOR_MODE=1

Format

Operation type: GET

URL: https://{ip}:{port}/metrics

Preferably set {ip} to {manage_ip} in Starting Coordinator. If the parameter is not configured, set {ip} to the value of the manage_ip parameter in the ms_coordinator.json configuration file.
The {port} field uses the value of the external_port parameter in the ms_coordinator.json configuration file.

Request Parameters

None

Usage Example

Request example:

GET https://{ip}:{port}/metrics

Response example:

# HELP request_received_total Number of requests received so far.
# TYPE request_received_total counter
request_received_total{model_name="llama3-8b"} 3188
# HELP request_success_total Number of requests proceed successfully so far.
# TYPE request_success_total counter
request_success_total{model_name="llama3-8b"} 2267
# HELP request_failed_total Number of requests failed so far.
# TYPE request_failed_total counter
request_failed_total{model_name="llama3-8b"} 0
# HELP num_preemptions_total Cumulative number of preemption from the engine.
# TYPE num_preemptions_total counter
num_preemptions_total{model_name="llama3-8b"} 637
# HELP num_requests_running Number of requests currently running on NPU.
# TYPE num_requests_running gauge
num_requests_running{model_name="llama3-8b"} 0
# HELP num_requests_waiting Number of requests waiting to be processed.
# TYPE num_requests_waiting gauge
num_requests_waiting{model_name="llama3-8b"} 0
# HELP num_requests_swapped Number of requests swapped to CPU.
# TYPE num_requests_swapped gauge
num_requests_swapped{model_name="llama3-8b"} 0
# HELP prompt_tokens_total Number of prefill tokens processed.
# TYPE prompt_tokens_total counter
prompt_tokens_total{model_name="llama3-8b"} 9564
# HELP generation_tokens_total Number of generation tokens processed.
# TYPE generation_tokens_total counter
generation_tokens_total{model_name="llama3-8b"} 84425
# HELP avg_prompt_throughput_toks_per_s Average prefill throughput in tokens/s.
# TYPE avg_prompt_throughput_toks_per_s gauge
avg_prompt_throughput_toks_per_s{model_name="llama3-8b"} 0.586739718914032
# HELP avg_generation_throughput_toks_per_s Average generation throughput in tokens/s.
# TYPE avg_generation_throughput_toks_per_s gauge
avg_generation_throughput_toks_per_s{model_name="llama3-8b"} 2.375296831130981
# HELP failed_request_perc Requests failure rate. 1 means 100 percent usage.
# TYPE failed_request_perc gauge
failed_request_perc{model_name="llama3-8b"} 0
# HELP npu_cache_usage_perc NPU KV-cache usage. 1 means 100 percent usage.
# TYPE npu_cache_usage_perc gauge
npu_cache_usage_perc{model_name="llama3-8b"} 1
# HELP cpu_cache_usage_perc CPU KV-cache usage. 1 means 100 percent usage.
# TYPE cpu_cache_usage_perc gauge
cpu_cache_usage_perc{model_name="llama3-8b"} 0
# HELP npu_prefix_cache_hit_rate NPU prefix cache block hit rate.
# TYPE npu_prefix_cache_hit_rate gauge
npu_prefix_cache_hit_rate{model_name="llama3-8b"} 0.5
# HELP time_to_first_token_seconds Histogram of time to first token in seconds.
# TYPE time_to_first_token_seconds histogram
time_to_first_token_seconds_count{model_name="llama3-8b"} 2523
time_to_first_token_seconds_sum{model_name="llama3-8b"} 9740.00200343132
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.001"} 0
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.005"} 0
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.01"} 0
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.02"} 0
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.04"} 0
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.06"} 10
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.08"} 54
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.1"} 104
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.25"} 256
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.5"} 256
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="0.75"} 276
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="1"} 321
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="2.5"} 628
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="5"} 1148
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="7.5"} 2523
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="10"} 2523
time_to_first_token_seconds_bucket{model_name="llama3-8b",le="+Inf"} 2523
# HELP time_per_output_token_seconds Histogram of time per output token in seconds.
# TYPE time_per_output_token_seconds histogram
time_per_output_token_seconds_count{model_name="llama3-8b"} 85800
time_per_output_token_seconds_sum{model_name="llama3-8b"} 4445.857012826018
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.01"} 0
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.025"} 0
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.05"} 3
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.075"} 12
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.1"} 40283
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.15"} 83145
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.2"} 83339
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.3"} 83339
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.4"} 83539
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.5"} 85139
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="0.75"} 85740
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="1"} 85800
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="2.5"} 85800
time_per_output_token_seconds_bucket{model_name="llama3-8b",le="+Inf"} 85800
# HELP e2e_request_latency_seconds Histogram of end to end request latency in seconds.
# TYPE e2e_request_latency_seconds histogram
e2e_request_latency_seconds_count{model_name="llama3-8b"} 2267
e2e_request_latency_seconds_sum{model_name="llama3-8b"} 12684.5319980979
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="1"} 27
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="2.5"} 268
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="5"} 712
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="10"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="15"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="20"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="30"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="40"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="50"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="60"} 2267
e2e_request_latency_seconds_bucket{model_name="llama3-8b",le="+Inf"} 2267
# HELP request_prompt_tokens Number of prefill tokens processed.
# TYPE request_prompt_tokens histogram
request_prompt_tokens_count{model_name="llama3-8b"} 3188
request_prompt_tokens_sum{model_name="llama3-8b"} 9564
request_prompt_tokens_bucket{model_name="llama3-8b",le="10"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="50"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="100"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="200"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="500"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="1000"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="2000"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="5000"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="10000"} 3188
request_prompt_tokens_bucket{model_name="llama3-8b",le="+Inf"} 3188
# HELP request_generation_tokens Number of generation tokens processed.
# TYPE request_generation_tokens histogram
request_generation_tokens_count{model_name="llama3-8b"} 2267
request_generation_tokens_sum{model_name="llama3-8b"} 84425
request_generation_tokens_bucket{model_name="llama3-8b",le="10"} 0
request_generation_tokens_bucket{model_name="llama3-8b",le="50"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="100"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="200"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="500"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="1000"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="2000"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="5000"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="10000"} 2267
request_generation_tokens_bucket{model_name="llama3-8b",le="+Inf"} 2267

Output Description

Status code 200: The query is successful. The message body is the cluster management and control metric summary in Prometheus metrics format.
Status code 503: Query fails.

The metric type and content are determined by the Server. The Coordinator is only responsible for summarizing metrics of each node in the cluster.

Parameter	Type	Description
request_received_total	Counter	Number of requests received by the server. model_name: user-defined model name, which is used in all example responses.
request_success_total	Counter	Number of requests that have been successfully executed by the server.
request_failed_total	Counter	Number of failed inference requests on the server.
num_requests_running	Gauge	Number of requests that are being executed on the server.
num_requests_waiting	Gauge	Number of requests to be scheduled on the server.
num_requests_swapped	Gauge	Number of requests that are currently swapped to the CPU by the server.
num_preemptions_total	Counter	Total number of request preemptions on the server.
prompt_tokens_total	Counter	Number of prefill tokens that have been processed.
generation_tokens_total	Counter	Number of generation tokens that have been processed.
avg_prompt_throughput_toks_per_s	Gauge	Latest average prefill throughput by the time when the last request is completed, in tokens/s.
avg_generation_throughput_toks_per_s	Gauge	Latest average generation throughput by the time when the last request is generated, in tokens/s.
failed_request_perc	Gauge	Failure rate of inference requests on the server. The value 1 indicates 100%.
npu_cache_usage_perc	Gauge	NPU usage of the current KV cache. The value 1 indicates 100%.
cpu_cache_usage_perc	Gauge	CPU memory usage of the current KV cache. The value 1 indicates 100%.
npu_prefix_cache_hit_rate	Gauge	Hit rate of the prefix cache on the NPU. The value 1 indicates 100%.
time_to_first_token_seconds	Histogram	Time to first token (TTFT), indicating the time consumed for generating the first inference token. The unit is second. The TTFT is the pure inference time, excluding the time for HTTP communication, tokenizer, and KV cache transmission in prefill-decode disaggregation. time_to_first_token_seconds_count: number of requests that have measured TTFT so far. time_to_first_token_seconds_sum: sum of the first token delays of all requests that have been completed and collected. time_to_first_token_seconds_bucket: TTFT of requests measured by histogram buckets so far. le: less than or equal to the boundary of a histogram bucket. NOTE: All the le parameters in the response example have the same meaning.
time_per_output_token_seconds	Histogram	Time Per Output Token (TPOT), which indicates the interval for generating two consecutive tokens, in seconds. The TPOT is the pure inference time, excluding the time for HTTP communication and KV cache transmission in prefill-decode disaggregation. time_per_output_token_seconds_count: number of tokens that have measured TPOT so far. time_per_output_token_seconds_sum: sum of TPOT of all tokens so far. time_per_output_token_seconds_bucket: TPOT measured by histogram buckets so far.
e2e_request_latency_seconds	Histogram	End-to-end latency, indicating how long it takes from receiving a request to executing it, in seconds. e2e_request_latency_seconds_count: number of requests that have measured end-to-end latency so far. e2e_request_latency_seconds_sum: sum of end-to-end latency of all requests so far. e2e_request_latency_seconds_bucket: end-to-end latency of requests measured by histogram buckets so far.
request_prompt_tokens	Histogram	Number of tokens input by a request, that is, number of tokens generated after the input prompt is processed by the tokenizer. request_prompt_tokens_count: number of requests that have queried current metrics so far. request_prompt_tokens_sum: sum of input tokens of all requests that have queried current metrics so far. request_prompt_tokens_bucket: number of input tokens measured by histogram buckets so far.
request_generation_tokens	Histogram	Number of tokens output by a request, that is, number of tokens generated after model inference. request_generation_tokens_count: number of requests that have queried current metrics so far. request_generation_tokens_sum: sum of output tokens of all requests that have queried current metrics so far. request_generation_tokens_bucket: number of output tokens measured by histogram buckets so far.

Parent topic: Intra-cluster Communication APIs