In foundation model training or inference scenarios, host-side performance jitter is one of the typical issues affecting model execution efficiency. It may cause operator dispatch latency in AI frameworks such as PyTorch, thereby slowing down model execution. The root cause lies in complex host-side software stack interactions, such as the Python interpreter, AI framework, and CANN software stack, which are difficult to accurately locate.
To address this issue, the Function Monitor tool is developed by integrating the lightweight Linux performance profiling capability of openEuler . This tool monitors function execution duration on the host, collects CPU PMU metrics (such as [object Object] and [object Object]) with low overhead during the execution of specified functions, and enables joint data analysis with Ftrace and MindStudio Profiler. This helps users efficiently identify and optimize host-side performance issues.
Procedure
- Use a Python decorator or
[object Object]statement to collect function execution duration and CPU PMU metrics. The tool supports data filtering based on execution duration thresholds and persistently writes the collected results to log files. - Process the collected log files and convert them into Chrome Trace JSON format.
- Import the data into for visualization to analyze the relationship between function execution duration and CPU PMU metrics.
- This tool is supported only in the PyTorch framework. Other AI frameworks are not supported.
- When you set the
[object Object]environment variable to[object Object]and collect CPU PMU metrics, ensure that the current user has root privileges. - You can use the
[object Object]decorator and[object Object]context manager provided in[object Object]together in different functions or code blocks to collect data. However, you must not stack them within the same function or code block. If a function is already decorated with[object Object]and you use[object Object]inside it, data collection may become abnormal.
- Build and install the
[object Object]Python wheel package. For details, see the . - Obtain the collection and conversion scripts and , and the file operation script provided in the repository.
You can collect data using either a Python function decorator or [object Object] statement. The following table describes the configuration of environment variables applicable to both methods.
When you set [object Object] to [object Object], you can specify the list of perf events to collect by using the [object Object] parameter of the [object Object] class in [object Object]. The default value is [object Object].
You can run the [object Object] command to view all supported perf events. For details, see the . You can select perf events as needed.
The [object Object] decorator provided in [object Object] embeds data collection logic into specified application functions to collect function execution duration and CPU PMU metrics.
Parameters
Example
Set environment variables.
[object Object]Modify the
[object Object]parameter of the[object Object]class in[object Object]to specify the perf events to collect. The default value is[object Object].[object Object]Apply the
[object Object]decorator in the PyTorch model script to functions that require data collection.[object Object]
Output Description
After data collection finishes, the tool generates a log file in the path specified by the [object Object] environment variable. By default, this is the [object Object] directory under the home directory of the current user. The file name format is [object Object], where [object Object] represents the process ID.
The [object Object] script also provides the [object Object] context manager. You can encapsulate data collection logic within a [object Object] statement block to collect execution duration and CPU PMU metrics for a specified code block.
Parameters
Example
Set environment variables.
[object Object]Modify the
[object Object]parameter of the[object Object]class in[object Object]to specify the perf events to collect. The default value is[object Object].[object Object]Import the
[object Object]context manager in the PyTorch model script, and encapsulate the target code block within a[object Object]statement block.[object Object]
Output Description
After data collection finishes, the tool generates a log file in the path specified by the [object Object] environment variable. By default, this is the [object Object] directory under the home directory of the current user. The file name format is [object Object], where [object Object] represents the process ID.
After data collection finishes, you can use the [object Object] script to convert the collected log files into Chrome Trace JSON format. This allows you to import them into MindStudio Insight for joint visualization and analysis.
Usage
Command-line Options
Example
Import the converted Chrome Trace JSON file into the MindStudio Insight visualization tool to view the function execution duration and collected CPU PMU metrics.
Each colored block represents a function execution event, containing the function name and its execution duration. The [object Object] section contains CPU PMU metrics collected during the execution of that function, such as page faults and LLC cache misses.
Due to underlying limitations of the [object Object] library, you must ensure that the current user has root privileges when calling decorators or interfaces defined in [object Object]. Otherwise, data collection might fail.
The data postprocessing script [object Object] has no special privilege requirements and can run with normal user privileges.