Analyzing Profile Data
Developers can identify performance bottlenecks by analyzing the timeline and summary files obtained by parsing the profile data with the Profiling tool.
The following describes key profile data files and analysis methods. For details about more profile data files, see Performance Tuning Tool User Guide .
- Timeline file: step_trace_*.csvThe step_trace_*.csv file records iteration trace data, including the duration of each iteration. The main fields are described as follows:
- Iteration Time: computation time of an iteration, including the time of the FP/BP and Grad Refresh phases.
- FP to BP Time: computation time of forward and backward propagation on the network.
- Iteration Refresh: iteration trailing time.
- Data Aug Bound: interval between two adjacent iterations.
The following figure shows a data sample. To analyze data, select a proper iteration ID and model ID.
Figure 1 Example of the step_trace_*.csv file
According to the preceding example, data of the model whose ID is 1 is obviously different from the subsequent data. This model is the initialization graph. Data of the model whose ID is 11 is the real iterative computational graph. Therefore, you need to select data of model 11 for analysis. In addition, when the model ID is 11 and iteration ID is 1, you can find that the Data Aug Bound time is long because compilation is performed in this phase. Therefore, you need to select data after iteration 2 of model 11 for analysis.
- Summary files: op_statistic_*.csv and op_summary_*.csv
The op_statistic_*.csv file records the AI Core and AI CPU operator execution times and time consumption. The op_summary_*.csv file records the detailed AI Core and AI CPU operator data.
Developers can preliminarily determine the time-consuming operators based on the op_statistic_*.csv file, and then search for details of the time-consuming operators in the op_summary_*.csv file to locate the minimum-granularity event.