Dynamic Profiling

Function

You can start or stop the profiling process at any time during the profiling. Dynamic profiling can be performed in launch or attach mode.

• Launch mode: When the msprof CLI starts dynamic profiling, it synchronously invokes the AI task and enters the interactive mode for profiling. You can run commands to start or stop profiling at any time.

  If the process started by the msprof --application command is the actual service process (for example, in msprof --application=main --dynamic=on, main is the actual service process), this mode is recommended.

• Attach mode: Start the AI task first. Then start msprof dynamic profiling and enter the interactive mode for profiling. You can run commands to start or stop profiling at any time.

  If the process started by the msprof --application command in launch mode is not the actual service process (for example, in msprof --application=run.sh --dynamic=on, run.sh starts the actual service process), you are advised to use the attach mode.

Precautions

• In launch mode, the environment variables PROFILING_MODE and PROFILING_OPTIONS cannot be set in the passed user application.
• In attach mode, set the environment variable PROFILING_MODE to dynamic, start the AI task, and then run the command.

  export PROFILING_MODE=dynamic

• In the same AI task, only one user is allowed to enter the interactive mode at a time.
• If the user does not start profiling within 30 minutes after entering the interactive mode, the system automatically exits the interactive mode. The user can run the corresponding command to enter the interactive mode again.
• In multi-rank (including cluster) scenarios, you are advised to use the attach mode to dynamically collect profile data. Profile data of each rank can only be collected independently.
• This function cannot be configured together with Delayed Profiling (the --delay and --duration options).
• If a user process is set to run in the background, the launch mode becomes invalid and the interaction interface cannot be accessed. In this case, you are advised to use the attach mode to dynamically collect profile data.
• If the service process is started in a container, you also need to run the msprof profiling command in the container.

Command Format

• Launch mode:

  msprof --dynamic=on [options] <app>
  --application=<app>

• Attach mode:

  msprof --dynamic=on [options]

  In launch mode, the user application must be passed. For details about the app parameters, see app Parameters. For details about the options parameters, see Table 1. Other items can also be added, for example, parameters in Profiling AI Tasks When They Are Running or Collecting AI processor System Data.

Command-line Options

ls ~/dynamic_profiling_socket_* 2>/dev/null | awk -F '_' '{print $4}' | paste -sd ',' -

Example

• Launch mode:

  msprof --dynamic=on --output=/home/projects/output  --runtime-api=on /home/projects/MyApp/out/main
  (msprof) start     
  ...
  (msprof) stop    
  ...
  (msprof) quit    
  ...

  The dynamical collection of profile data requires the passing of a user application.

• Attach mode:

  msprof --dynamic=on --pid=<pid> --output=/home/projects/output --runtime-api=on
  (msprof) start     
  ...
  (msprof) stop    
  ...
  (msprof) quit    
  ...

Output Description

The PROF_XXX directory is generated in the directory specified by --output. You are advised to use the parsing command to parse and export profile data.

During dynamic profiling, if multiple PIDs are configured, the process of each PID generates its own PROF_XXX directory in the directory specified by --output. In this case, you are advised to set --output to the parent directory of PROF_XXX during parsing so that multiple PROF_XXX directories can be parsed at the same time.

Obtaining the PID of an AI Task

You can run the command below to view the PID that can be collected in the current environment, and then perform the following operations to obtain the PID:

ls ~/dynamic_profiling_socket_* 2>/dev/null | awk -F '_' '{print $4}' | paste -sd ',' -

• AI task running on a physical machine

  Run the following command to view the PID of the AI task:

  npu-smi info

  Below is the command output. The value of Process id is the PID of the AI task.

  1 2 3 4 5

  +---------------------------+---------------+----------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===========================+===============+====================================================+ | 0 0 | 830236 | python3 | 58 | +===========================+===============+====================================================+

• AI task running in a privileged container

  In this scenario, the PID specified by --pid for dynamic profiling must be the PID of the AI task running in the privileged container.

  Perform the following steps to view the PID of the AI task running in the privileged container:

  1. Run the following command to view the PID of the AI task running on the host:

     npu-smi info

     

     The npu-smi info command can display only the PID of the AI task running on the host, regardless of whether the command is executed in the privileged container or on the host of the privileged container.

     Below is the command output. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.

     1 2 3 4 5

     +---------------------------+---------------+----------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===========================+===============+====================================================+ | 0 0 | 837666 | | 58 | +===========================+===============+====================================================+

  2. Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:

     cat /proc/837666/status | grep NSpid

     The command output is as follows:

     NSpid:	837666	849

     In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

• AI task running in a non-privileged container
  • Method 1: Run the following command in the non-privileged container view to view the PID of the AI task running in the non-privileged container:

    npu-smi info

    Below is the command output. The value of Process id is the PID of the AI task running in the non-privileged container.

    +---------------------------+---------------+----------------------------------------------------+
    | NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
    +===========================+===============+====================================================+
    | 0       0                 | 849       |                          | 58                      |
    +===========================+===============+====================================================+

  • Method 2: Run the following command in the host view of the non-privileged container to view the PID of the AI task running on the host:

    npu-smi info

    Below is the command output. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.

    +---------------------------+---------------+----------------------------------------------------+
    | NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
    +===========================+===============+====================================================+
    | 0       0                 | 837666        |                          | 58                      |
    +===========================+===============+====================================================+

    Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:

    cat /proc/837666/status | grep NSpid

    The command output is as follows:

    NSpid:	837666	849

    In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

In multi-device scenarios, you are advised to run AI tasks on each device one by one and obtain the PID in the preceding mode. If AI tasks are running on multiple devices at the same time, obtain any PID for profiling. Currently, multiple PIDs cannot be specified for dynamic profiling at the same time.