[object Object][object Object]

In current AI training and inference scenarios, Python multithreading is widely used for tasks such as operator dispatch and memory movement. However, due to Python's Global Interpreter Lock (GIL), the interpreter can execute only one thread at a time in a multithreaded environment and cannot achieve true parallel execution. Threads must contend for the GIL, which leads to frequent lock contention, unbalanced scheduling, and reduced overall model performance.

To address this issue, the GIL Tracer tool is developed by integrating openEuler . It provides a simple and non-intrusive mechanism for GIL behavior monitoring, enabling the analysis of GIL usage in Python multithreaded programs. This helps developers identify performance bottlenecks, optimize thread scheduling strategies, and perform joint analysis with MindStudio Profiler to improve overall application performance.

Procedure

1. Collect Python GIL data using either the command line or APIs.
2. Process the collected GIL data and convert it into Chrome Trace JSON format.
3. Import the converted data into together with Profiler data for visualization and analysis. For Profiler data collection, see section "Performance Tuning Tools" in or .

[object Object]

1. Install [object Object]. For details, see the .

2. Add the [object Object] binary to the [object Object] environment variable.

   [object Object]

3. Configure [object Object] environment variables, including:

   1. Runtime dependencies. For details, see the .
   2. Logging and data directory configurations. For details, see the .

4. Obtain the collection and conversion scripts [object Object] and [object Object], and the file utility script [object Object] provided in the repository. Synchronous collection of Profiler and GIL data is recommended.

[object Object]

You can collect GIL data using either the command line or APIs.

[object Object]

Description

Collect GIL data by executing the [object Object] script.

Syntax

Command-line Options

[object Object]undefined

Examples

Start a Python model process on the NPU and obtain its PID, such as [object Object]. Open a new terminal window to run the [object Object] script (requires root privileges).

• Example 1: Collect GIL data for a specified process with PID [object Object] for 10 seconds.

  [object Object]

• Example 2: Collect GIL data for all Python processes running on the NPU for 10 seconds.

  [object Object]

• Example 3: Collect GIL data for a specified process with PID [object Object] and stop data collection manually.

  [object Object]

Output Description

After data collection finishes, the tool generates the corresponding GIL data files in the data directory configured by [object Object] (default: [object Object]). The file name format is [object Object], where [object Object] represents the process ID and [object Object] represents the Rank ID of the model process. The default value of the Rank ID is [object Object].

[object Object]

Description

The [object Object] script provides APIs to independently control the start and stop of GIL data collection. Developers can embed the collection logic precisely into applications. This is suitable for scenarios requiring dynamic control over collection timing, conditional triggers, or deep integration with business workflows.

Example

1. API for starting data collection

   [object Object]

2. API for stopping data collection

   [object Object]

The code sample is as follows:

Parameters

[object Object]undefined

Output Description

After data collection finishes, the tool generates the corresponding GIL data files in the data directory configured by [object Object] (default: [object Object]). The file name format is [object Object], where [object Object] represents the process ID and [object Object] represents the Rank ID of the model process. The default value of the Rank ID is [object Object].

[object Object]

Description

The [object Object] script processes the collected GIL data and converts it into Chrome Trace JSON format. This allows you to import it into MindStudio Insight for joint visualization and analysis.

Syntax

Command-line Options

[object Object]undefined

Example

[object Object]

Import the converted Chrome Trace JSON file into MindStudio Insight to view the collected GIL data.

Independent Collection

Each colored block represents a GIL event, which belongs to one of three types:

• [object Object]: indicates that the thread is waiting for the GIL lock.
• [object Object]: indicates that the thread holds the GIL lock.
• [object Object]: indicates that the thread releases the GIL lock.

Joint Visualization with Profiler Data

When you use APIs to collect GIL and Profiler data simultaneously, MindStudio Insight can display both datasets together. This facilitates the analysis and optimization of Python multithreading performance issues.

[object Object]

Due to underlying limitations of the [object Object] tool, you must ensure that the current user has root privileges when executing the [object Object] script or calling its APIs. Otherwise, data collection might fail.

The data postprocessing script [object Object] has no special privilege requirements and can run with normal user privileges.