文档

Data Storing Directories

The structure of the directory for storing raw profile data is as follows:

  • Directory structure when the tensorboard_trace_handler function is called:
    • If the step ID in the kernel_details.csv file is empty, you can view the step information of the operator in the trace_view.json file or collect profile data again.
    • The following data is sampled based on the actual environment. If the corresponding conditions do not exist in the environment, the corresponding data or file will not be generated. For example, if the model does not have the AI CPU operator, the corresponding data_preprocess.csv file will not be generated even if the sampling is performed.
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
    └── localhost.localdomain_139247_20230628101435_ascend_pt    // Parsing result directory, whose format is {worker_name}_{timestamp}_ascend_pt. The default value of {worker_name} is {hostname}_{pid}.
    ├── profiler_info.json              // In multi-device or cluster scenarios, the naming rule is profiler_info_{Rank_ID}.json, which is used to record Profiler-related metadata.
    ├── profiler_metadata.json
    ├── ASCEND_PROFILER_OUTPUT          // Profile data collection with Ascend PyTorch Profiler
    │   ├── ascend_pytorch_profiler_{rank_id}.db          // When export_type is set to torch_npu.profiler.ExportType.Db, a .db file is generated in the directory. Other .json and .csv files are not generated.
    │   ├── analysis.db                // When export_type is set to torch_npu.profiler.ExportType.Db in multi-device or cluster scenarios where communication is involved, a .db file is generated in the directory, and it is displayed by the MindStudio Insight tool. Other .json and .csv files are not generated.
    │   ├── communication.json         // Visualized data foundation for performance analysis in multi-device or cluster scenarios where communication is involved. It is generated by configuring profiler_level to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2 in experimental_config.
    │   ├── communication_matrix.json   // Basic information file of small communication operators, which is generated by configuring profiler_level to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2 in experimental_config.
    │   ├── data_preprocess.csv        // Generated by configuring profiler_level to torch_npu.profiler.ProfilerLevel.Level2 in experimental_config.
    │   ├── kernel_details.csv
    │   ├── l2_cache.csv               // Generated by configuring l2_cache to True in experimental_config.
    │   ├── memory_record.csv
    │   ├── npu_module_mem.csv
    │   ├── operator_details.csv
    │   ├── operator_memory.csv
    │   ├── step_trace_time.csv        // Computation and communication time statistics in iterations.
    │   ├── op_statistic.csv           // AI Core and AI CPU operator calling times and time consumption.
    │   ├── api_statistic.csv          // Generated when profiler_level is configured to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2 in experimental_config.
    │   └── trace_view.json
    ├── FRAMEWORK                      // Raw profile data on the framework side, which can be ignored. Delete this directory when data_simplification is set to True.
    └── PROF_000001_20230628101435646_FKFLNPEPPRRCFCBA  // Profile data at the CANN layer, which is named in the format of PROF_{number}_{timestamp}_{character string}. When data_simplification is set to True, only the raw profile data in this directory is retained, and other data is deleted.
          ├── analyze      // Generated when profiler_level is configured to torch_npu.profiler.ProfilerLevel.Level1 or torch_npu.profiler.ProfilerLevel.Level2 in experimental_config.
          ├── device_*
          ├── host
          ├── mindstudio_profiler_log
          └── mindstudio_profiler_output
├── localhost.localdomain_139247_20230628101435_ascend_pt_op_args // Directory for storing the operator statistics file, which is generated by configuring record_op_args to True in experimental_config.

    The Ascend PyTorch Profiler API associates and integrates the framework data with the CANN Profiling data to form profile data files such as trace, kernel, and memory. Files are stored in the ASCEND_PROFILER_OUTPUT directory, including Timeline and Summary Data (in JSON and CSV formats).

    The FRAMEWORK directory stores the raw profile data on the framework side and can be ignored. The PROF directory stores the profile data collected by CANN Profiling, which is mainly stored in the mindstudio_profiler_output directory. For details about the data, see Profile Data File References.

  • When the export_chrome_trace method is called, the Ascend PyTorch Profiler API writes the parsed trace data to a *.json file. * indicates the file name. If the file does not exist, it is automatically created in the specified path.