Profile Data Parsing and Export
Applicability
Product |
Supported |
|---|---|
Atlas 350 Accelerator Card |
√ |
√ |
|
√ |
|
√ |
|
√ |
|
√ |
Function
This function is used to parse and export profile data.
Precautions
- Complete the operations in Before You Start first.
- The profiling has completed.
- Direct parsing on the device is not supported for the products below. You must copy the generated PROF_XXX directory to an environment where the Toolkit package is installed.
Ascend RC scenario of theAtlas 200I/500 A2 inference product
Command Format
msprof --export=on --output=<dir> [--type=<type>] [--reports=<reports_sample_config.json>] [--iteration-id=<number>] [--model-id=<number>] [--summary-format=<csv/json>] [--clear=on]
Command-line Options
Option |
Description |
Optional/Required |
|---|---|---|
--export |
Profile data parsing and export. The value can be on or off (default).
To export data of a specific model (model ID) or iteration (iteration ID), run the msprof --export command again to configure the --model-id and --iteration-id options after the msprof profiling command is executed. The PROF_XXX files that are not parsed are automatically parsed and then exported. Example: msprof --export=on --output=/home/profiler_data/PROF_XXX |
Required |
--output |
Directory for storing the profile data file. The value must be the PROF_XXX directory or the parent directory of the PROF_XXX directory, for example, /home/profiler_data/PROF_XXX. The path cannot contain the following characters: "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "|", "\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", "\\`". |
Required |
--type |
Format of the profile data parsing result file. That is, you can choose the format of the result file generated after the profile data collected by the msprof command is automatically parsed. The available formats include:
The default value is text. |
Optional |
--reports |
After the custom reports_sample_config.json configuration file is passed, the corresponding profile data files are exported based on the range specified in the configuration file. For details, see Example (--reports). For the Atlas 350 Accelerator Card, this parameter is not supported currently. |
Optional |
--iteration-id |
Iteration ID. The value must be a positive integer. The default value is 1. This option and --model-id must be configured at the same time.
|
Optional |
--model-id |
Model ID. The value must be a positive integer. This option and --iteration-id must be configured at the same time.
|
Optional |
--summary-format |
Export format of a summary data file. The values are as follows:
Supported only when --type=text. |
Optional |
--python-path |
Path to the Python interpreter used for parsing. The Python version must be 3.7.5 or later. If the default Python version in the environment is earlier than 3.7.5, use this parameter to set the Python version to 3.7.5 or later. If this option is executed by a user with higher permissions, do not specify a path with low permissions. |
Optional |
--clear |
Data clearance mode. After this option is enabled, the sqlite directory in PROF_XXX/device_{id} is deleted after profile data is exported, so as to save storage space. The value can be on or off (default). |
Optional |
- By default, all profile data is exported.
- In single-operator scenarios or scenarios where only AI processor system data is collected (that means the --application option is not specified in the msprof profiling command), the --iteration-id and --model-id options are not supported.
Example
Specify the /home/profiler_data/PROF_XXX directory as the profile data directory and run the following command:
msprof --export=on --output=/home/profiler_data/PROF_XXX
Example (--reports)
Specify the /home/profiler_data/PROF_XXX directory as the profile data directory, pass the custom reports_sample_config.json configuration file, and run the following command:
msprof --export=on --output=/home/profiler_data/PROF_XXX --reports=${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/reports_sample_config.json
Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann.
The path cannot contain the following characters: "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "|", "\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", "\\`".
- --reports specifies the reports_sample_config.json file. Use this option with the --export option and configure --type=text. It controls only the timeline data in the .json file. The summary data in the .csv file will always be fully exported.
- Soft links are not supported. The maximum file size is 64 MB, and the maximum length of the file path (including the file name) is 1,024 characters.
By default, the reports_sample_config.json file is stored in the ${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/ directory. The content of the file is as follows:
The reports_sample_config.json file can be created in any directory, on which you have read and write permissions.
{
"json_process": {
"ascend": true,
"acc_pmu": true,
"cann": true,
"ddr": true,
"stars_chip_trans": true,
"hbm": true,
"communication": true,
"hccs": true,
"os_runtime_api": true,
"network_usage": true,
"disk_usage": true,
"memory_usage": true,
"cpu_usage": true,
"msproftx": true,
"npu_mem": true,
"overlap_analyse": true,
"pcie": true,
"sio": true,
"stars_soc": true,
"step_trace": true,
"freq": true,
"llc": true,
"nic": true,
"roce": true,
"qos": true,
"device_tx": true
}
}
The preceding parameters are switches for profiling specific data. You can set them to true or false (or delete unwanted fields). The following timeline data layers in the msprof_*.json file can be profiled: CANN, Ascend Hardware, AI Core Freq, On-Chip Memory, Communication, Overlap Analysis, and NPU_MEM.
- The prerequisite for exporting these data is that the original profile data already contains these fields, which means these data fields are profiled.
- Ensure that the reports_sample_config.json file is in the correct format. Otherwise, the following situations may occur:
- If the file contains errors like misspellings, the --reports option fails, exporting all profile data instead.
- The system stops the export and reports an error if the --reports option cannot access the configuration file due to insufficient permissions or a non-existent file.
Output Description
After the msprof --export command is executed, the mindstudio_profiler_output directory is generated in the PROF_XXX directory.
The structure of the generated profile data directory is as follows:
- Single-process profiling
1 2 3 4 5 6 7 8 9 10 11 12 13 14
└── PROF_XXX ├── device_0 │ └── data ├── device_1 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt
- Multi-process profiling
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
└── PROF_XXX1 ├── device_0 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt └── PROF_XXX2 ├── device_1 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt
- msprof_*.db is a .db file that summarizes all profile data. The .json files in the mindstudio_profiler_output directory are timeline information files, which collect the running durations of operators and tasks and display the collected data in color blocks. The .csv files are summary information files, which summarize the running durations in tables. For details about profile data, see Profile Data File References.
- In multi-device setups, if single-process collection is started, only one PROF_XXX directory is generated. If multi-process collection is started, multiple PROF_XXX directories are generated. In addition, the device directory is generated in the PROF_XXX directory. The number of device directories generated in each PROF_XXX directory is related to the actual user operations and does not affect profile data analysis.
- The files in the mindstudio_profiler_output directory are generated based on the actual profile data. If the necessary profile data file is absent, the corresponding timeline and summary data will be unavailable.
- For a msprof process that is forcibly interrupted, the tool saves the collected raw profile data. You can run msprof --parse to re-parse the data and then run msprof --export.