Function

You can start or stop the profiling process at any time during the profiling. Dynamic profiling can be performed in launch or attach mode.

Launch mode: When the msprof CLI starts dynamic profiling, it synchronously invokes the AI task and enters the interactive mode for profiling. You can run commands to start or stop profiling at any time.
If the process started by the msprof --application command is the actual service process (for example, in msprof --application=main --dynamic=on, main is the actual service process), this mode is recommended.
Attach mode: Start the AI task first. Then start msprof dynamic profiling and enter the interactive mode for profiling. You can run commands to start or stop profiling at any time.
If the process started by the msprof --application command in launch mode is not the actual service process (for example, in msprof --application=run.sh --dynamic=on, run.sh starts the actual service process), you are advised to use the attach mode.

Precautions

In launch mode, the environment variables PROFILING_MODE and PROFILING_OPTIONS cannot be set in the passed user application.
In attach mode, set the environment variable PROFILING_MODE to dynamic, start the AI task, and then run the command.
```
export PROFILING_MODE=dynamic
```
In the same AI task, only one user is allowed to enter the interactive mode at a time.
If the user does not start profiling within 30 minutes after entering the interactive mode, the system automatically exits the interactive mode. The user can run the corresponding command to enter the interactive mode again.
In multi-device (including cluster) scenarios, the attach mode is recommended for dynamic profile data collection.
This function cannot be configured together with Delayed Profiling (--delay and --duration options).
If a user process is set to run in the background, the launch mode becomes invalid and the interaction interface cannot be accessed. In this case, you are advised to use the attach mode to dynamically collect profile data.
If the service process is started in a container, you also need to run the msprof profiling command in the container.

Command Format

Launch mode:

msprof --dynamic=on [options] <app>
--application=<app>

Attach mode:

msprof --dynamic=on --pid=<pid> [options]

In launch mode, the user program must be passed. For details about app, see app Parameters. For details about options, see Table 1. Other profiling options can also be added, for example, parameters in Profiling AI Tasks When They Are Running or Profiling System Data of the AI processor.

Command-line Options

**Table 1** Options
Option	Optional/Required	Description
--dynamic	Required	Switch that controls dynamic profile data collection, either on or off (default).
--pid	Required for attach mode; not required for launch mode	PID of the application to profile. You can configure one or more PIDs at a time, for example, --pid=1,2,3. For details about how to obtain the PID, see Obtaining the PID of an AI Task. You can also run the following command to obtain all PIDs that can be profiled in the current environment: ls ~/dynamic_profiling_socket_* 2>/dev/null \| awk -F '_' '{print $4}' \| paste -sd ',' -
--output	Optional	Path for storing profile data. By default, this parameter is not configured, indicating that profile data is saved in the current path. The path cannot contain the following characters: "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "\|", "\\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", and "\\`".
start	Optional	Starts collection. NOTE: You can run the start and stop commands up to 100 times combined. Exceeding this limit will cause the server to disconnect, limiting profile samples to 50. When the server is reconnected, the count will be reset. Repeatedly using start and stop commands might halt the profiling process with the stop command, stopping CANN's data reporting. This triggers an ERROR log, but this is expected behavior.
stop	Optional	Stops collection. Each time the start and stop commands are executed, a PROF_XXX directory for storing the data file is generated in the path specified by the --output option.
quit	Optional	Stops collection and exits the interactive mode. After quit is executed, the AI task continues to run normally. You can run the msprof dynamic profiling command again to re-enter the interactive mode and restart dynamic profile data collection.

Example

Launch mode:

msprof --dynamic=on --output=/home/projects/output  --runtime-api=on /home/projects/MyApp/out/main
(msprof) start     
...
(msprof) stop    
...
(msprof) quit    
...

The dynamical collection of profile data requires the passing of a user application.

Attach mode:

msprof --dynamic=on --pid=<pid> --output=/home/projects/output
(msprof) start     
...
(msprof) stop    
...
(msprof) quit    
...

Output Description

The PROF_XXX directory is generated in the directory specified by --output. You are advised to use the parsing command to parse and export profile data.

During dynamic profiling, if multiple PIDs are configured, the process of each PID generates its own PROF_XXX directory in the directory specified by --output. In this case, you are advised to set --output to the parent directory of PROF_XXX during parsing so that multiple PROF_XXX directories can be parsed at the same time.

Obtaining the PID of an AI Task

You can run the command below to view the PID that can be collected in the current environment, and then perform the following operations to obtain the PID:

ls ~/dynamic_profiling_socket_* 2>/dev/null | awk -F '_' '{print $4}' | paste -sd ',' -

AI task running on a physical machine

Run the following command to view the PID of the AI task:

npu-smi info

Below is the command output. The value of Process id is the PID of the AI task.

+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 830236        | python3                  | 58                      |
+===========================+===============+====================================================+

AI task running in a privileged container

In this scenario, the PID specified by --pid for dynamic profiling must be the PID of the AI task running in the privileged container.

Perform the following steps to view the PID of the AI task running in the privileged container:

Run the following command to view the PID of the AI task running on the host:

npu-smi info

The npu-smi info command can display only the PID of the AI task running on the host, regardless of whether the command is executed in the privileged container or on the host of the privileged container.

Below is the command output. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.

+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 837666        |                          | 58                      |
+===========================+===============+====================================================+

Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:
```
cat /proc/837666/status | grep NSpid
```
The command output is as follows:
```
NSpid:	837666	849
```
In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

AI task running in a non-privileged container

Method 1: Run the following command in the non-privileged container view to view the PID of the AI task running in the non-privileged container:

npu-smi info

Below is the command output. The value of Process id is the PID of the AI task running in the non-privileged container.

+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 849       |                          | 58                      |
+===========================+===============+====================================================+

Method 2: Run the following command in the host view of the non-privileged container to view the PID of the AI task running on the host:
```
npu-smi info
```
Below is the command output. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.
```
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 837666        |                          | 58                      |
+===========================+===============+====================================================+
```
Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:
```
cat /proc/837666/status | grep NSpid
```
The command output is as follows:
```
NSpid:	837666	849
```
In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

In multi-device scenarios, you are advised to run AI tasks on each device one by one and obtain the PID in the preceding mode. If AI tasks are running on multiple devices at the same time, obtain any PID for profiling. Currently, multiple PIDs cannot be specified for dynamic profiling at the same time.