Profile Data Collection

Prerequisites

You have performed operations in Environment Setup.
You have performed the PyTorch-related operations in Model Development and Migration and obtained the GPU and Ascend NPU environments that can properly execute training jobs.

Before collecting profile data, delete the accuracy data collection APIs in Accuracy Data Collection from the training script pytorch_main.py because accuracy data collection and profile data collection cannot be performed at the same time.

Performing Collection

Add the Ascend PyTorch Profiler to the training script pytorch_main.py in the Ascend NPU environment.

You can copy the complete code from Code Sample for Using the Ascend PyTorch Profiler API to Collect Profile Data and execute it directly. The following examples only show where to add the tool API in the script.

 23
import torch_npu
from torch_npu.contrib import transfer_to_npu
 26
...
   model.train()

   end = time.time()
   experimental_config = torch_npu.profiler._ExperimentalConfig(
       profiler_level=torch_npu.profiler.ProfilerLevel.Level0,
       data_simplification=False)
   with torch_npu.profiler.profile(
       activities=[
           torch_npu.profiler.ProfilerActivity.CPU,
           torch_npu.profiler.ProfilerActivity.NPU
           ],
       schedule=torch_npu.profiler.schedule(wait=0, warmup=0, active=1, repeat=1, skip_first=1),
       on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./profiling_data"),
       experimental_config=experimental_config) as prof:
       for i, (images, target) in enumerate(train_loader):
...
           # measure elapsed time
           batch_time.update(time.time() - end)
           end = time.time()
358
           prof.step()
...

For details about how to configure more comprehensive collection parameters and corresponding APIs, see ""Ascend PyTorch Profiler"" in Profiling Instructions.
Profile data occupies certain disk space. As a result, the server may be unavailable if the disk space is used up. Profile data size depends on model parameters, profiling settings, and the iteration count. Make sure the storage location has enough free disk space.

Run the training script command. The tool collects the profile data during model training.
```
python pytorch_main.py -a resnet50 -b 32 --gpu 1 --dummy
```

View the result file of profile data collected during PyTorch training.

After the training is complete, the profile data result directory of the Ascend PyTorch Profiler API is generated in the path specified by the torch_npu.profiler.tensorboard_trace_handler API. The following gives an example.

└── msprof_1784298_20250620085947065_ascend_pt
    ├── ASCEND_PROFILER_OUTPUT
    │   ├── ascend_pytorch_profiler_{Rank_ID}.db
    │   ├── kernel_details.csv
    │   ├── operator_details.csv
    │   ├── step_trace_time.csv
    │   └── trace_view.json
    ├── FRAMEWORK
...
    ├── PROF_000001_20250620085947066_FLRBJLNFMBIDRPMB
    │   ├── device_1
    │   │   ├── data
...
    │   ├── host
    │   │   ├── data
...
    │   ├── mindstudio_profiler_log
...
    │   ├── msprof_*.db
    │   └── mindstudio_profiler_output
    │       ├── api_statistic_20250620085954.csv
    │       ├── msprof_20250620085953.json
    │       ├── op_summary_20250620085954.csv
    │       ├── README.txt
    │       └── task_time_20250620085954.csv
    ├── profiler_info.json
    └── profiler_metadata.json

You can use the msprof-analyze tool in MSTT to analyze the profile data collected by the Ascend PyTorch Profiler API, or use MindStudio Insight to perform visualized analysis. For details, see Using msprof-analyze to Analyze Profile Data and Using MindStudio Insight to Visualize Profile Data.

Parent topic: Model Performance Tuning