Dynamic Profiling

Instructions

Profile data can be dynamically collected in launch or attach mode.

• Launch mode: When the msprof command line starts dynamic collection, it synchronously invokes the AI tasks and enters the interactive mode for profile data collection. You can run commands to start or stop collection at any time.
• Attach mode: You start the AI task first, and then start msprof dynamic collection and enter the interactive mode for profile data collection. You can run commands to start or stop collection at any time.

• If a user process is set to run in the background, the launch mode becomes invalid and the interaction interface cannot be accessed. In this case, you are advised to use the attach mode to dynamically collect profile data.
• You are advised to perform operations in Parsing and Exporting Profile Data to parse and export the profile data in the PROF_XXX directory.
• If a service process is started in a container, you also need to run the msprof command in the container.

Availability

Atlas 200/500 A2 Inference Product

Atlas Inference Series Product

Atlas Training Series Product

Atlas A2 Training Series Product/Atlas 800I A2 Inference Product

Atlas A3 Training Series Product

Precautions

• Before using this mode to collect profile data, ensure that operations in Before You Start have been completed.
• Before dynamically collecting profile data, ensure that your AI task can run properly in the operating environment.
• In attach mode, you need to set the required environment variable before starting a training job.

  export PROFILING_MODE=dynamic

Restrictions

• In the same AI task, only one user is allowed to enter the interactive mode at a time.
• If the user does not start profiling within 30 minutes after entering the interactive mode, the system automatically exits the interactive mode. The user can run the corresponding command to enter the interactive mode again.
• In multi-rank (including cluster) scenarios, you are advised to use the attach mode to dynamically collect profile data. Profile data of each rank can only be collected independently.
• Do not configure this function with both --delay and --duration options described in Delayed Profiling.
• In launch mode, the environment variables PROFILING_MODE and PROFILING_OPTIONS cannot be set in the passed user application.

Command Example

Use either of the following modes:

• Launch mode:

  msprof --dynamic=on --output=/home/projects/output --model-execution=on --runtime-api=on --aicpu=on /home/projects/MyApp/out/main
  > start
      
  ...
  > stop
     
  ...
  > quit
     
  ...

  The dynamic collection of profile data requires the passing of a user application.

• Attach mode:

  msprof --dynamic=on --pid=<pid> --output=/home/projects/output --model-execution=on --runtime-api=on --aicpu=on
  > start
      
  ...
  > stop
     
  ...
  > quit
     
  ...

Options

• You can run the start and stop commands up to 100 times combined. Exceeding this limit will cause the server to disconnect, limiting profile samples to 50. When the server is reconnected, the count will be reset.
• Repeatedly using start and stop commands might halt the profiling process with the stop command, stopping CANN's data reporting. This triggers an ERROR log, but this is expected behavior.

Obtaining the PID of an AI Task

• AI task running on a physical machine

  Run the following command to view the PID of the AI task:

  npu-smi info

  The command output is as follows. The value of Process id is the PID of the AI task.

  1 2 3 4 5

  +---------------------------+---------------+----------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===========================+===============+====================================================+ | 0 0 | 830236 | python3 | 58 | +===========================+===============+====================================================+

• AI task running in a privileged container

  In this scenario, the PID specified by --pid for dynamic profiling must be the PID of the AI task running in the privileged container.

  Perform the following steps to view the PID of the AI task running in the privileged container:

  1. Run the following command to view the PID of the AI task running on the host:

     npu-smi info

     

     The npu-smi info command can display only the PID of the AI task running on the host, regardless of whether the command is executed in the privileged container or on the host of the privileged container.

     The command output is as follows. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.

     1 2 3 4 5

     +---------------------------+---------------+----------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===========================+===============+====================================================+ | 0 0 | 837666 | | 58 | +===========================+===============+====================================================+

  2. Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:

     cat /proc/837666/status | grep NSpid

     The command output is as follows:

     NSpid: 837666 849

     In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

• AI task running in a non-privileged container
  • Method 1: Run the following command in the non-privileged container view to view the PID of the AI task running in the non-privileged container:

    npu-smi info

    The command output is as follows. The value of Process id is the PID of the AI task running in the non-privileged container.

    +---------------------------+---------------+----------------------------------------------------+
    | NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
    +===========================+===============+====================================================+
    | 0       0                 | 849       |                          | 58                      |
    +===========================+===============+====================================================+

  • Method 2: Run the following command in the host view of the non-privileged container to view the PID of the AI task running on the host:

    npu-smi info

    The command output is as follows. The value of Process id is the PID of the AI task running on the host. The following uses PID 837666 as an example.

    +---------------------------+---------------+----------------------------------------------------+
    | NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
    +===========================+===============+====================================================+
    | 0       0                 | 837666        |                          | 58                      |
    +===========================+===============+====================================================+

    Run the following command in the host view to view the PID mapping between the AI task on the host and that in the privileged container:

    cat /proc/837666/status | grep NSpid

    The command output is as follows:

    NSpid: 837666 849

    In the preceding command output, 849 is the PID of the AI task running in the privileged container. The PID specified by --pid for dynamic collection must be 849.

In multi-device scenario setups, you are advised to run AI tasks on each device one by one and obtain the PID in the preceding mode. If AI tasks are running on multiple devices at the same time, obtain any PID for profiling. Currently, multiple PIDs cannot be specified for dynamic profiling at the same time.

Data Parsing

You are advised to perform operations in Parsing and Exporting Profile Data to parse and export profile data in the PROF_XXX directory.