Profile Data Collection

Prerequisites

You have performed operations in Environment Setup.
You have completed the training jobs in the MindSpore scenario as instructed in Model Development and Migration.

Before collecting profile data, delete APIs in Accuracy Data Collection from the training script mindspore_main.py because accuracy data collection and profile data collection cannot be performed at the same time.

Performing Collection

The following uses MindSpore 2.7.0 as an example. If a performance comparison is required, you can also collect the profile data from MindSpore 2.6.0.

Add the MindSpore Profiler API tool to the training script mindspore_main.py in the Ascend NPU environment.

You can copy the complete code from Code Sample for Using MindSpore Profiler APIs to Collect Profile Data and execute it directly. The following examples only show where to add the tool API in the script.

 ...
from mindspore.profiler import ProfilerLevel, ProfilerActivity, AicoreMetrics
...
if __name__ == "__main__":
    mindspore.set_context(mode=mindspore.PYNATIVE_MODE)
    mindspore.set_device("Ascend")

    # Init Profiler
    experimental_config = mindspore.profiler._ExperimentalConfig(
        profiler_level=ProfilerLevel.Level0,
        aic_metrics=AicoreMetrics.AiCoreNone,
        l2_cache=False,
        mstx=False,
        data_simplification=False,
    )

    step = 0
    # Note that the Profiler should be initialized before model.train
    with mindspore.profiler.profile(
            activities=[ProfilerActivity.CPU, ProfilerActivity.NPU],
            schedule=mindspore.profiler.schedule(
                wait=0, warmup=0, active=1, repeat=1, skip_first=0
            ),
            on_trace_ready=mindspore.profiler.tensorboard_trace_handler("./profiling_data"),
            profile_memory=False,
            experimental_config=experimental_config,
    ) as prof:
        # Train Model
        for data, label in ds.GeneratorDataset(generator_net(), ["data", "label"]):
            train_step(data, label)
            print(f"train step {step}")
            step += 1
            prof.step()
        print("train finish")

For details about corresponding APIs and how to configure more comprehensive collection parameters, see mindspore.profiler.profile.
Profile data occupies certain disk space. As a result, the server may be unavailable if the disk space is used up. Profile data size depends on model parameters, profiling settings, and the iteration count. Make sure the storage location has enough free disk space.

Run the training script command. The tool collects the profile data during model training.
```
python mindspore_main.py
```

View the result file of profile data collected during MindSpore training.

After the training is complete, the profile data result directory of the MindSpore Profiler API is generated in the directory specified by the mindspore.profiler.tensorboard_trace_handler API. The following gives an example.

└── msprof_1198203_20250617064811905_ascend_ms
├── ASCEND_PROFILER_OUTPUT
│   ├── api_statistic.csv
│   ├── dataset.csv
│   ├── kernel_details.csv
│   ├── step_trace_time.csv
│   └── trace_view.json
├── FRAMEWORK
...
├── PROF_000001_20250617064812088_QGGJHHNAODRDOJFB
│   ├── device_0
│   │   ├── data
...
│   ├── host
│   │   ├── data
...
│   ├── mindstudio_profiler_log
...
│   └── mindstudio_profiler_output
│       ├── api_statistic_20250617064814.csv
│       ├── msprof_20250617064813.json
│       ├── op_summary_20250617064814.csv
│       ├── README.txt
│       └── task_time_20250617064814.csv
└── profiler_info_0.json

You can use the msprof-analyze tool in MSTT to analyze the profile data collected by the MindSpore Profiler API, or use MindStudio Insight to perform visualized analysis. For details, see Using msprof-analyze to Analyze Profile Data and Using MindStudio Insight to Visualize Profile Data.

Parent topic: Model Performance Tuning