Parsing and Exporting Profile Data
For the following products, profile data cannot be directly parsed on devices. Instead, you need to copy the generated PROF_XXX directory to an environment where the CANN Toolkit package and ops operator package are installed for data parsing and export.
- Ascend RC scenario of the Atlas 200/500 A2 Inference Product
Prerequisites
- Ensure that operations in Before You Start have been completed.
- The profile data has been collected.
Procedure
Run the data exporting command.
Example:
msprof --export=on --output=<dir> [--type=<type>] [--reports=<reports_sample_config.json>] [--iteration-id=<number>] [--model-id=<number>] [--summary-format=<csv/json>] [--clear=on]
Option |
Description |
Required/Optional |
|---|---|---|
--export |
Profile data parsing and export, either on or off (default). To export data of a specific model (model ID) or iteration (iteration ID), run the msprof --export command again to configure the --model-id and --iteration-id options after the msprof profiling command is executed. The PROF_XXX files that are not parsed are automatically parsed and then exported. Example: msprof --export=on --output=/home/HwHiAiUser |
Required |
--output |
Directory for storing the profile data file. The value must be the parent directory of the PROF_XXX or PROF_XXX directory, for example, /home/HwHiAiUser/profiler_data/PROF_XXX. The following special characters are not allowed in the path: "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "|", "\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", "\\`". |
Required |
--type |
Format of the profile data parsing result file. That is, you can choose the format of the result file generated after the profile data collected by the msprof command is automatically parsed. The available formats include:
The default value is text. |
Optional |
--reports |
Pass a custom reports_sample_config.json file to export profile data files according to the range specified in the file. For details, see Parameter Usage of --reports. |
Optional |
--iteration-id |
Iteration ID. The value must be a positive integer. The default value is 1. This option and --model-id must be configured at the same time.
|
Optional |
--model-id |
Model ID. The value must be a positive integer. This option and --iteration-id must be configured at the same time.
|
Optional |
--summary-format |
Export format of a summary data file. The values are as follows:
Supported only when --type=text. |
Optional |
--python-path |
Path to the Python interpreter used for parsing. The Python version must be 3.7.5 or later. |
Optional |
--clear |
Data simplification mode. After this option is enabled, the sqlite directory in PROF_XXX/device_{id} is deleted after profile data is exported, so as to save storage space. The value can be on or off (default). |
Optional |
Note 1: all profile data is exported by default. Note 2: In single-operator scenarios and scenarios where only Profiling the Ascend AI Processor System is involved, --iteration-id and --model-id are not supported. |
||
After the preceding command is executed, the mindstudio_profiler_output directory is generated in the PROF_XXX directory under --output.
The structure of the generated profile data directory is as follows:
- Single-process collection
1 2 3 4 5 6 7 8 9 10 11 12 13 14
└── PROF_XXX ├── device_0 │ └── data ├── device_1 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt
- Multi-process collection
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
└── PROF_XXX1 ├── device_0 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt └── PROF_XXX2 ├── device_1 │ └── data ├── host │ └── data ├── msprof_*.db └── mindstudio_profiler_output ├── msprof_{timestamp}.json ├── step_trace_{timestamp}.json ├── xx_*.csv ... └── README.txt
- msprof_*.db is a .db file that summarizes all profile data. The .json files in the mindstudio_profiler_output directory are timeline files, which collect the running durations of operators and tasks and display the collected data in color blocks. The .csv files are summary files, which summarize the running durations in tables. For details about profile data, see Profile Data File References.
- In multi-device setups, if single-process collection is started, only one PROF_XXX directory is generated. If multi-process collection is started, multiple PROF_XXX directories are generated. In addition, the device directory is generated in the PROF_XXX directory. The number of device directories generated in each PROF_XXX directory is related to the actual user operations and does not affect profile data analysis.
- The files in the mindstudio_profiler_output directory are generated based on the actual collected profile data. If the necessary profile data file is absent, the corresponding timeline and summary data will be unavailable.
- For a msprof process that is forcibly interrupted, the tool saves the collected raw profile data. You can run msprof --parse to re-parse the data and then run msprof --export.
Parameter Usage of --reports
Command example:
msprof --export=on --output=./ --reports=${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/reports_sample_config.json
Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.
The following special characters are not allowed in the path: "\n", "\\n", "\f", "\\f", "\r", "\\r", "\b", "\\b", "\t", "\\t", "\v", "\\v", "\u007F", "\\u007F", "\"", "\\\"", "'", "\'", "\\", "\\\\", "%", "\\%", ">", "\\>", "<", "\\<", "|", "\\|", "&", "\\&", "$", "\\$", ";", "\\;", "`", "\\`".
- --reports specifies the reports_sample_config.json file. Use this option with the --export option and configure --type=text. It controls only the timeline data in the .json file. The summary data in the .csv file will always be fully exported.
- Soft links are not supported. The maximum file size is 64 MB, and the maximum length of the file path (including the file name) is 1,024 characters.
By default, the reports_sample_config.json file is stored in the ${INSTALL_DIR}/tools/profiler/profiler_tool/analysis/msconfig/ directory. The content of the file is as follows:
The reports_sample_config.json file can be created in any directory, on which you have read and write permissions.
{
"json_process": {
"ascend": true,
"acc_pmu": true,
"cann": true,
"ddr": true,
"stars_chip_trans": true,
"hbm": true,
"communication": true,
"hccs": true,
"os_runtime_api": true,
"network_usage": true,
"disk_usage": true,
"memory_usage": true,
"cpu_usage": true,
"msproftx": true,
"npu_mem": true,
"overlap_analyse": true,
"pcie": true,
"sio": true,
"stars_soc": true,
"step_trace": true,
"freq": true,
"llc": true,
"nic": true,
"roce": true,
"qos": true,
"device_tx": true
}
}
The preceding parameters are switches for profiling specific data. You can set them to true or false( or delete unwanted fields). The following timeline data layers in the msprof_*.json file can be profiled: CANN, Ascend Hardware, AI Core Freq, On-Chip Memory, Communication, Overlap Analysis, and NPU_MEM
- The prerequisite for exporting these data is that the original profile data already contains these fields, which means these data fields are profiled.
- Ensure that the reports_sample_config.json file is in the correct format. Otherwise, the following situations may occur:
- If the file contains errors like misspellings, the --reports option fails, exporting all profile data instead.
- The system stops the export and reports an error if the --reports option cannot access the configuration file due to insufficient permissions or a non-existent file.