Exporting Profile Data

Applicability

Product	Supported
Atlas 350 Accelerator Card	√
Atlas A3 training product/Atlas A3 inference product	√
Atlas A2 training product/Atlas A2 inference product	√
Atlas 200I/500 A2 inference product	√
Atlas inference product	√
Atlas training product	√

Function

Exports profile data.

Precautions

Before exporting profile data, complete the operations in Parsing Profile Data.

Command Format

Exporting timeline data and .db files

python3 msprof.py export timeline -dir <dir> [-reports <reports_sample_config.json>] [--iteration-id <iteration_id>] [--model-id <model-id>] [--iteration-count <iteration_count>] [--clear]

Exporting summary data and .db files

python3 msprof.py export summary -dir <dir> [--iteration-id <iteration_id>] [--model-id <model-id>] [--iteration-count <iteration_count>] [--format <export_format>] [--clear]

Exporting .db files
```
python3 msprof.py export db -dir <dir>
```

Command-line Options

**Table 1** Command-line options
Option	Description	Optional/Required
-dir, or --collection-dir	Directory of collected profile data. The value must be PROF_XXX or the parent directory of PROF_XXX. Example: /home/HwHiAiUser/profiler_data/PROF_XXX	Required
-reports	After the custom reports_sample_config.json configuration file is passed, the corresponding profile data files are exported based on the range specified in the configuration file. This parameter is implemented the same way as msprof --reports. For details, see Example (--reports).	Optional
--iteration-id	Iteration ID. The value must be a positive integer. The default value is 1. This option and --model-id must be configured at the same time. For the Atlas A2 training product/Atlas A2 inference product, if --model-id is set to 4294967295, this option specifies the iteration ID whose data is collected each step. The iteration ID increases by 1 each time a step is executed. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. For the Atlas A3 training product/Atlas A3 inference product, if --model-id is set to 4294967295, this option specifies the iteration ID whose data is collected each step. The iteration ID increases by 1 each time a step is executed. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. If --model-id is set to other values, this option specifies the iteration ID for graph-based statistics collection. (The iteration ID is incremented by 1 each time a graph is executed. When a script is compiled into multiple graphs, the iteration ID is different from the step ID at the script layer.)	Optional
--model-id	Model ID. The value must be a positive integer. This option must be specified in combination with --iteration-id. If neither --iteration-id nor --model-id is specified, data for all model IDs is exported by default. For the Atlas A2 training product/Atlas A2 inference product, --model-id can be set to 4294967295, which specifies the step mode. That is, the value of --iteration-id specifies parsing by step. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. For the Atlas A3 training product/Atlas A3 inference product, --model-id can be set to 4294967295, which specifies the step mode. That is, the value of --iteration-id specifies parsing by step. Only profile data of the MindSpore framework (version 2.3 or later) can be parsed. If --model-id is set to other values, the graph mode is used. That is, the value of --iteration-id specifies parsing by graph.	Optional
--iteration-count	Number of consecutive steps for which data will be exported. The value is an integer ranging from 1 to 5. The value of --iteration-id is the start step. For example, if --iteration-count is set to 3 and --iteration-id is set to 1, data for steps 1, 2, and 3 will be exported.	Optional
--format	File format, either csv (default) or json. This option is supported only when the summary parameter is configured. The following uses summary files in .csv format as examples.	Optional
--clear	Data clearance mode. After this option is enabled, the sqlite directory in PROF_XXX/device_{id} is deleted after profile data is exported, so as to save storage space. When this parameter is configured, the data clearance mode is enabled. This parameter is not configured by default.	Optional
-h or --help	Help information.	Optional

Example

Log in to the environment where the Toolkit package is installed.
Switch to the directory where the msprof.py script is located.
${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msprof. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
Export profile data. The timeline, summary, and db files can be exported. The command formats are as follows:
- Exporting timeline data and .db files
```
python3 msprof.py export timeline -dir /home/HwHiAiUser/profiler_data/PROF_XXX
```
- Exporting summary data and .db files
```
python3 msprof.py export summary -dir /home/HwHiAiUser/profiler_data/PROF_XXX
```
- Exporting .db files: Generate a .db file (msprof_timestamp.db) that summarizes all profile data.
```
python3 msprof.py export db -dir /home/HwHiAiUser/profiler_data/PROF_XXX
```
- By default, all profile data is exported.
- In single-operator scenarios or scenarios where only AI processor system data is collected (that means the --application option is not specified in the msprof profiling command), the --iteration-id and --model-id options are not supported.

Output Description

After the preceding command is executed, find the mindstudio_profiler_output directory and the msprof_*.db file in the PROF_XXX directory under collection-dir.

The following shows the directory structure of the generated profile data:

Single-process collection

└── PROF_XXX
      ├── device_0
      │    └── data
      ├── device_1
      │    └── data
      ├── host
      │    └── data
      ├── msprof_*.db
      └── mindstudio_profiler_output
            ├── msprof_{timestamp}.json
            ├── step_trace_{timestamp}.json
            ├── xx_*.csv
             ...
            └── README.txt

Multi-process collection

└── PROF_XXX1
      ├── device_0
      │    └── data
      ├── host
      │    └── data
      ├── msprof_*.db
      └── mindstudio_profiler_output
            ├── msprof_{timestamp}.json
            ├── step_trace_{timestamp}.json
            ├── xx_*.csv
             ...
            └── README.txt
└── PROF_XXX2
      ├── device_1
      │    └── data
      ├── host
      │    └── data
      ├── msprof_*.db
      └── mindstudio_profiler_output
            ├── msprof_{timestamp}.json
            ├── step_trace_{timestamp}.json
            ├── xx_*.csv
             ...
            └── README.txt

In multi-device setups, if single-process collection is started, only one PROF_XXX directory is generated. If multi-process collection is started, multiple PROF_XXX directories are generated. In addition, the device directory is generated in the PROF_XXX directory. The number of device directories generated in each PROF_XXX directory is related to the actual user operations and does not affect profile data analysis.
For details about profile data, see Profile Data File References.
The files in the mindstudio_profiler_output directory are generated based on the actual profile data. If the necessary profile data file is absent, the corresponding timeline and summary data will be unavailable.
You can run the export command to directly export summary reports from the profile data parsing result. Even the profile data has not been parsed, the export command can parse the profile data and export the reports.
For a msprof collection process that is forcibly interrupted, the tool saves the collected raw profile data. You can also run export to parse and export the data.

Parent topic: Using the msprof.py Script to Parse, Query, and Export Profile Data