This metric monitoring tool enhances the native monitoring capability of the vLLM-Ascend inference framework. While vLLM-Ascend provides basic metrics out of the box, this tool adds the following capabilities:
- KVCache monitoring: tracks total blocks, idle block count, and the number of blocks allocated per DP domain.
- Token and throughput: monitors the number of input/output tokens and total tokens per DP domain.
- Custom metric: adds a timer metric for the execution duration of any function.
[object Object]
- In the Ascend environment, install the matching CANN Toolkit and ops operator packages, and configure CANN environment variables. For details, see .
- Install vLLM and vLLM-Ascend. Verify that vLLM-ascend can run properly and the metrics endpoint is accessible. For details, see .
- Build the .run package from the source code and upgrade the tool. For details, see the section Upgrade in .
- Version compatibility: Ensure that vLLM-Ascend, CANN, and collection tool versions meet the requirements in the Appendix.
- Resource usage: Data monitoring requires enabling Prometheus multi-process mode (
[object Object]). This may impact inference performance. - Function restrictions: Some advanced features may require specific vLLM-Ascend versions.
Follow these steps to complete the metric monitoring process:
- Set environment variables and start the service (with Prometheus multi-process mode enabled).
- Enable the collection function by modifying the
[object Object]field in the configuration file (independent of the[object Object]field). - Send an inference request.
- View metrics by accessing the metrics endpoint or Grafana.
Before starting the inference service, set the following environment variables:
[object Object]is the collection configuration file (shared with performance profiling).[object Object]controls Prometheus metric collection, and[object Object]controls performance profiling. If the file does not exist, the default configuration is automatically generated. For details, see .[object Object]configures symbols. For details about how to customize symbols, see [Symbol Configuration User Guide](#Symbol Configuration User Guide).- 8000 is the default port for vLLM serving inference startup. This document uses port 8000 as an example. To change the serving startup port, you can use the
[object Object]command line parameter to specify a port when starting the vLLM service. For details, see .
In this tool, [object Object] in the [object Object] configuration file controls metric collection. [object Object] is independent of the [object Object] field (which controls performance profiling), so you can enable them separately or together.
Enable metric collection by setting [object Object] in [object Object] to [object Object]. If the field is not set (default) or does not exist, it is set to [object Object], metric collection is disabled. If the JSON file does not contain the [object Object] field, manually add it and change the value to [object Object].
The modification takes effect without restarting the service. When the function is enabled or disabled, the corresponding information (for example, "Metric collection enabled" or "Metric collection disabled") is recorded in the log.
Send an inference request to generate monitoring data.
Obtain data through the metrics endpoint of the vLLM service.
You can also configure Prometheus to scrape metrics from this endpoint and visualize the metrics using Grafana.
The symbol configuration file defines the functions/methods to be monitored, and supports flexible configuration and custom attribute collection.
- When the collection symbols are updated, you need to restart the vLLM service to load the new configuration.
- All custom metric names are automatically prefixed with
[object Object]and must comply with .
You can access the metrics endpoint (for example, [object Object]) of the vLLM service to view the current metrics. The following is an example output of the custom [object Object] ([object Object] type), which can be visualized using visualization tools like Grafana.
The following are metric symbols built into the tool. All metric names are automatically prefixed with [object Object] and contain the [object Object] label. (The scheduling process is the actual DP domain ID, and the request process is [object Object].)
[object Object]