Error Message "Incorrect schedule" Is Displayed During Data Collection with Ascend PyTorch Profiler

Symptom

When Ascend PyTorch Profiler APIs are used to collect PyTorch profile data, the message "Incorrect schedule" is displayed, as shown in the following figure.

profiler.py: Incorrect schedule: Stop profiler while current state is WARMUP which will result in empty parsed data.

profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.

profiler.py: Stop profiler while current state is RECORD_AND_SAVE, perhaps the scheduling cycle has not yet completed.

Possible Cause

• Scenario 1

  The schedule parameter is improperly set. As a result, Profiler exits in advance before the set scheduling period ends. See the following case.

  

  The actual model training step is 1, but skip_first=1 and active=2 are set in schedule. At this time, Profiler is in the RECORD state (ready to collect data), but the training process has already exited, resulting in missing or empty profile data.

• Scenario 2

  If the schedule parameter repeat is set to the default value 0, the collected data of the last step may be incomplete, and the log records profiler.py: Stop profiler while current state is RECORD_AND_SAVE, perhaps the scheduling cycle has not yet completed. and profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data. In this case, you are not advised to use the data of the last step as the reference for profile data analysis.

Troubleshooting

Ensure that the schedule is correctly configured (you are advised to use this formula: total number of steps >= skip_first + (wait + warmup + active) × repeat) and ensure that there are sufficient steps for profile data collection after Profiler completes the schedule.