This document describes the profiling tool (msServiceProfiler) for inference serving. The msServiceProfiler APIs collect start and end times for key processes, identify main functions or iterations, and record important events during the MindIE Motor inference serving. msServiceProfiler profiles various data and helps pinpoint performance issues quickly.
The tool usage process is as follows:
[object Object]
[object Object]
The hardware environment supported by the tool is the same as that supported by MindIE. For details, see "Installing MindIE" in .
- Install the matching CANN Toolkit and ops operator packages, and configure CANN environment variables. For details, see .
- Install .
- Install and configure MindIE and ensure that MindIE Motor can run properly. For details, see .
- After the preceding environment is prepared, you can use to check environment variables and serving configurations.
Function description[object Object][object Object]
Collects serving profile data.
Precautions[object Object][object Object]
- Enabling the
[object Object]feature in msServiceProfiler simultaneously with the msProf dynamic collection feature may cause conflicts. It is advised to use them separately. For details about the msProf dynamic collection feature, see .
Examples[object Object][object Object]
Create a configuration file for profiling. The tool profiles serving data based on settings in a JSON file that defines whether to profile data and where to store it.
Automatic creation: The file can be automatically created. After the
[object Object]environment variable is configured in , running MindIE Motor can automatically create a JSON file with the default settings.Manual creation: The JSON configuration file can be created in any path. The following uses the
[object Object]file as an example. The configuration file format is as follows:[object Object]Table 1 Parameter description
[object Object]undefined
[object Object][object Object]Collect data.
Configure environment variables and specify the collection configuration file
[object Object].[object Object]- If the JSON file does not exist: A default configuration file is automatically created with
[object Object]set to[object Object](collection disabled). After starting MindIE Motor, follow to set[object Object]to[object Object]and start collection. - If the file already exists: No file is created—the existing configuration is used.
- If the JSON file does not exist: A default configuration file is automatically created with
Run MindIE Motor.
[object Object][object Object]Start the collection task.
Open another CLI window. You can modify the
[object Object]field in[object Object]to enable or disable the profiling in real time. Enabling or disabling collection generates corresponding logs. See below for details.After the collection is complete, the profile data is saved to the path specified by the
[object Object]parameter in[object Object].
[object Object]
Output Description[object Object][object Object]
During serving profile data collection, logs are generated to indicate the status of the collection process. You can control logging using the [object Object] environment variable as described below.
The [object Object] environment variable configures the log level for data collection. The following is an example:
Available log levels (default: [object Object]):
[object Object]: includes information such as whether profiling is enabled and where the data is saved.The following is an example:
[object Object]DEBUG: detailed logs. Adds to INFO-level logs: configuration file path, NPU/CPU collection status, and more.
The following is an example:
[object Object]WARNING: Adds to INFO-level logs: parameter errors, dynamic library load failures, and other warnings.
[object Object]ERROR: Only error logs are printed.
The following is an example:
[object Object]
Function Description[object Object][object Object]
Parses serving profile data.
Precautions[object Object][object Object]
- During data collection, files that have been written to disk are locked. Parsing can only be started after collection is complete; otherwise, the error "db is lock" will be displayed.
- Do not enable dynamic collection again before data parsing is complete, as this may degrade collection performance.
- The following versions are required for data parsing:
- python >= 3.10
- pandas >= 2.2
- numpy >= 1.24.3
- psutil >= 5.9.5
- matplotlib >= 3.7.5
- scipy >= 1.7.2
Syntax[object Object][object Object]
For details about the options parameter, see .
Parameter Description[object Object][object Object]
Table 1 Parameter description
Examples[object Object][object Object]
Run the following command to perform basic parsing on profile data.
In addition to basic parsing, you can perform on profile data by different dimensions (request-level, batch-level, overall service-level). You can also perform fine-grained on data of different batches.
Output Description[object Object][object Object]
After the parsing command is executed, the are output to the path specified by the [object Object] parameter.
The results of [object Object] are as follows:
Table 1 Mapping between collection domains and parsed results[object Object][object Object]
[object Object]
These files include:
It is an SQLite database file for generating visualized line charts.
It contains the following database tables, with their specific purposes described below:
Table 2 profiler.db
For details about PD disaggregation deployment and related concepts, see "Cluster Service Deployment > PD Disaggregation Service Deployment" in . The file connects to Grafana for image display during visualization. It does not include detailed descriptions of its entries.
Records trace data of inference service requests. You can use different visualization tools to view the data. For details, see .
Records detailed batch-level data for inference serving.
Table 3 batch.csv
[object Object]
It is automatically generated in the speculative inference scenario and records the speculative inference data for each request at each time step.
Table 3-2 spec_decode.csv
[object Object]
Records memory usage during inference.
Table 4 kvcache.csv
Records detailed request-level data for inference serving.
Table 5 request.csv
Records detailed data during the model forward execution in serving inference.
Table 6 forward.csv
Contains communication data for PD disaggregation scenarios. PD disaggregation is a multi-node, multi-device (cluster) deployment that requires a shared configuration file during .
For details about PD disaggregation deployment and related concepts, see "Cluster Service Deployment > PD Disaggregation Service Deployment" in .
Table 7 pd_split_communication.csv
Records KV cache transfer between P-nodes and D-nodes during PD disaggregation inference. PD disaggregation is a multi-node, multi-device (cluster) deployment that requires a shared configuration file during .
For details about PD disaggregation deployment and related concepts, see "Cluster Service Deployment > PD Disaggregation Service Deployment" in .
Table 8 pd_split_kvcache.csv
Records changes in the number of requests distributed to each node during PD disaggregation inference. PD disaggregation is a multi-node, multi-device (cluster) deployment that requires a shared configuration file during .
For details about PD disaggregation deployment and related concepts, see "Cluster Service Deployment > PD Disaggregation Service Deployment" in .
Table 9 coordinator.csv
Records the request status at each moment during serving inference (the number of [object Object], [object Object], or [object Object] requests). This data can be used to generate line charts that visualize request status trends over time.
Table 12 request_status.csv
A [object Object] is the minimum performance monitoring unit in distributed tracing, corresponding to a basic step in the inference process. The [object Object] directory stores profile data for different span types. By default, it contains [object Object] for forward inference data and [object Object] for batch scheduling data. Multiple span types can be customized. Exported data includes fields such as execution duration, host, and process ID. The following uses [object Object] as an example.
Table 13 BatchSchedule.csv
Records load imbalance analysis results for the GroupedMatmul operator, collected via MSPTI during DeepSeek expert model serving inference.
If data for [object Object] analysis exists, running the parsing command will generate a heatmap in the output directory by default. In the heatmap, the x-axis represents the process ID of each device, and the y-axis represents the decoder layer of the model. Brighter pixels indicate longer duration. Greater color variation across rows indicates more significant load imbalance.
Table 10 ep_balance.csv
Figure 1 ep_balance.png[object Object][object Object]
Records fast/slow rank analysis results for the MoeDistributeCombine and MoeDistributeDispatch operators, collected via MSPTI during DeepSeek expert model serving inference.
If data for [object Object] exists, running the parsing command will generate a box plot in the output directory by default. In the box plot, the x-axis represents the process ID of each device, and the y-axis represents the total duration. The plot displays the mean and the 2.5th/97.5th percentiles of the total duration. Greater disparity between devices (wider percentile intervals) indicates more significant fast/slow rank issues.
Table 11 moe_analysis.csv
Figure 2 moe_analysis.png[object Object][object Object]
This is an expert hotspot heatmap. In , pixel brightness indicates hotspot intensity (see the colorbar on the right), that is, brighter means higher heat.
[object Object]indicates the device name.[object Object]indicates the number of load balancing table updates during the serving profiling period when dynamic load balancing is enabled on MindIE. If disabled,[object Object].
Figure 3 Heatmap[object Object][object Object]
The x-axis represents expert ID, and the y-axis represents the model's MoE layer.
In the model instance, expert IDs are sorted by [object Object] in ascending order, then numbered sequentially within each rank. For example, with 16 cards each having 17 experts, expert ID 42 corresponds to [object Object] (the eighth expert) on [object Object] (the third card).
This is an expert hotspot heatmap. In , pixel brightness indicates hotspot intensity (see the colorbar on the right), that is, brighter means higher heat.
[object Object]indicates the device name.[object Object]indicates the number of load balancing table updates during the serving profiling period when dynamic load balancing is enabled on MindIE. If disabled,[object Object].
Figure 4 Heatmap[object Object][object Object]
The x-axis represents [object Object], and the y-axis represents the model's MoE layer.
This is an expert hotspot heatmap. In , pixel brightness indicates hotspot intensity (see the colorbar on the right), that is, brighter means higher heat.
[object Object]indicates the device name.[object Object]indicates the number of load balancing table updates during the serving profiling period when dynamic load balancing is enabled on MindIE. If disabled,[object Object].- This figure is generated only when the dynamic load balancing feature of MindIE is enabled.
Figure 5 Heatmap[object Object][object Object]
The x-axis represents the model expert IDs, where the shared experts are listed at the end. The y-axis represents the model's MoE layer.
This is an expert load imbalance line chart. shows the degree of expert load imbalance over time.
Figure 6 Expert load imbalance line chart[object Object][object Object] 
The horizontal coordinate tokens num indicates the number of model inference rounds, and the vertical coordinate balance ratio indicates the model load imbalance degree, which is calculated based on the standard deviation of expert heat.
The red dotted line indicates a change in the expert load balancing table. The corresponding x-axis value represents the system's local time, which is averaged across devices due to time drift. For accurate results, ensure all devices have synchronized local time before data collection.
MindStudio Insight can visualize the profile data () collected and analyzed by the service profiler. Currently, it supports visualizing [object Object] and [object Object] files. For detailed operations and visualization results, see the section "Serving Tuning" in .
Chrome tracing can visualize data from t[object Object] file collected and analyzed by the service profiler. To use it:
Enter [object Object] in the Chrome address bar, drag the .json file to the blank area to open it, and press the shortcut keys ([object Object]: zoom in; [object Object]: zoom out; [object Object]: move left; [object Object]: move right) to view the file.
[object Object]
Function Description [object Object][object Object]
Grafana can visualize profile data collected and analyzed by the service profiler.
Data Preparation [object Object][object Object]
has been completed, and have been generated.
Ensure that the SQLite database file [object Object] exists in the path specified by [object Object].
Environment Dependencies[object Object][object Object]
Tool versions: Grafana 11.3.0, and SQLite plugin 11.3.0.
Installing and Connecting Grafana
Download the official open-source version from . Extract and run: Example:
When configuring the Windows proxy, you need to add the Linux device IP prefix (for example, [object Object]). Access Grafana at [object Object]. Default username and password are both [object Object].
Figure 1 Grafana[object Object][object Object]
Example[object Object][object Object]
Create a data source, as shown in .
Figure 2 Data source[object Object][object Object]
Set the data source type to
[object Object], as shown in .Figure 3 Add data source[object Object][object Object]
Connect the generated SQLite database file
[object Object]to Grafana and record the[object Object], as shown in .Figure 4 Data sources[object Object][object Object]
Create a dashboard and import the line chart.
The visualization file
[object Object]is located at /xxx/Ascend/cann-{version}/tools/msserviceprofiler/python/ms_service_profiler/views/. Modify the datasource uid in the JSON file to the uid recorded in the previous step.[object Object]
Figure 5 uid[object Object][object Object]
[object Object]
Figure 6 json[object Object][object Object]
Create a new dashboard, paste the modified JSON content, and import it. Find the dashboard under
[object Object]with the specified name.Figure 7 Dashboards[object Object][object Object]
Figure 8 Import dashboard[object Object][object Object]
Figure 9 Setting parameters[object Object][object Object] parameter 
Visualization Results[object Object][object Object]
The generated Grafana dashboard contains the following visualized charts:
Table 1 Visualized charts
Batch_Size_curve
Line chart showing request counts per batch during batch scheduling.
x-axis: shows the batch number in the execution order, beginning with 0
y-axis: records the batch sizes for both prefill and decode batches separately.
Figure 10 Batch_Size_curve[object Object][object Object]
Request_Status_curve
Line chart showing sizes of requests in different states over time.
x-axis: timeline of the inference serving.
Vertical axis: size of the queue in the current state.
Figure 11 Request_Status_curve[object Object][object Object]
Kvcache_usage_percent_curve
Line chart showing the KV cache usage of all requests over time.
x-axis: timeline of the inference serving.
y-axis: KV cache usage of all requests Unit: %
Figure 12 Kvcache_usage_percent_curve[object Object][object Object]
First_Token_Latency_curve
Line chart showing the token latency across all requests over time
x-axis: timeline of the inference serving.
y-axis: average TTFT across all requests, along with its 99th, 90th, and 50th percentiles, and the minimum value (min), in μs.
Figure 13 First_Token_Latency_curve[object Object][object Object]
Prefill_Generate_Speed_Latency_curve
Line chart showing the average per-token latency of tokens being processed during the prefill phase across all requests over time.
x-axis: timeline of the inference serving.
y-axis: the average per-token latency during the prefill phase across all requests at each moment, along with its 99th, 90th, and 50th percentiles. Unit: tokens/s
Figure 14 Prefill_Generate_Speed_Latency_curve[object Object][object Object]
Decode_Generate_Speed_Latency_curve
Line chart showing the average per-token latency of tokens being processed during the decode phase across all requests over time.
x-axis: timeline of the inference serving.
y-axis: the average per-token latency during the decode phase across all requests at each moment, along with its 99th, 90th, and 50th percentiles. Unit: tokens/s
Figure 15 Decode_Generate_Speed_Latency_curve[object Object][object Object]
Request_Latency_curve
Line chart showing the end-to-end latency across all requests over time.
x-axis: timeline of the inference serving.
Vertical axis: the average value of end-to-end latency across all requests, along with its 99th, 90th, and 50th percentiles, in μs.
Figure 16 Request_Latency_curve[object Object][object Object]
The MindIE Motor framework includes built-in profiling code. This operation is optional.
To customize performance data collection, modify the serving framework's collection code following the examples below. For available APIs, see or .
[object Object]
Span data profiling:
Event data profiling:
Metric data profiling:
Link data profiling:
Attribute data profiling:
Different versions and frameworks of the foundation model inference service may perform differently. The serving profile data comparison tool uses the msServiceProfiler tool to analyze profile data and quickly spot potential issues. For details, see .
The tool uses Ascend-vLLM for profiling. It combines msServiceProfiler's data parsing and visualization features to help debug and improve vLLM inference and serving. For details, see .
Based on the profile data collected by the msServiceProfiler, this tool automatically optimizes serving parameters and test tool parameters. For details, see .