Profiling with Environment Variables
Data collection with environment variables applies to training/online inference of the TensorFlow framework. Unlike the collection mode using the TensorFlow framework API, the environment variable mode is to directly insert the PROFILING_OPTIONS environment variable into the training/online inference script to configure profiling items.
The --task-trace option will be discarded in later versions. Use --task-time instead.
Prerequisites
- Training scenario:
- Prepare a model trained on TensorFlow 1.15 and a matched dataset, and port the model to the AI processor. For details, see TensorFlow 1.15 Model Porting.
- Prepare a model trained on TensorFlow 2.x and a matched dataset, and port the model to the AI processor. For details, see TensorFlow 2.6.5 Model Porting.
- Online inference scenario: Download a pre-trained model and prepare the online inference script.
Procedure
export PROFILING_MODE=true
export PROFILING_OPTIONS='{"output":"/tmp/profiling","training_trace":"on","task_trace":"on","fp_point":"","bp_point":"","aic_metrics":"PipeUtilization"}'
For details about PROFILING_OPTIONS, see Profiling Options.
If PROFILING_MODE is set to true but PROFILING_OPTIONS is not set, training_trace, task_trace, hccl, aicpu, and aic_metrics (PipeUtilization) are profiled by default, and the profiled data is saved to the directory where the current AI task is located. If PROFILING_MODE is set to true and PROFILING_OPTIONS is set, see Profiling Options for the default settings of PROFILING_OPTIONS.
Profiling Results
After configuring PROFILING_OPTIONS, parse the raw data file and export the parsing results as a visualized profile data file by referring to Using the msprof Command to Parse, Query, and Export Profile Data. Save the file in the PROF_XXX/mindstudio_profiler_output directory.