Performance Analysis in Cluster Training Scenarios

Scenario

A cluster consists of multiple nodes, which are management in a unified manner on the management page. Each node has an independent system. In cluster scenarios, the tool collects profile data of each node, generates a PROF_XXX directory on each node, and pre-parses and summarizes all PROF_XXX directories to OBS. You need to manually copy all PROF_XXX directories summarized by OBS to an environment where the cluster data can be displayed and analyzed.

Currently, the following tool supports cluster data display and analysis: MindStudio Insight.

Profiling Process

The following figure shows the overall process of profile data collection.

Figure 1 Profiling Process

Environment Setup

• In cluster scenarios, you need to set up the environment by yourself.
• Install a proper CANN software package on the corresponding node as required. For details, see the CANN Software Installation.
• Install MindStudio Insight. For details, see MindStudio Insight User Guide.

Restrictions

In cluster scenarios, profile data of a maximum of 128 nodes can be collected. If eight devices are configured for each node, profile data of a maximum of 1024 devices can be collected.

Profiling

After the environment is set up, you can collect profile data in the cluster scenario as follows:

• Use msprof Profiling Commands to collect profile data.
• Use the Ascend PyTorch Profiler APIs to collect PyTorch profile data.
  1. Set up a distributed training environment and prepare the distributed training script used after migration. For details, see Model Porting in the PyTorch Model Porting and Tuning Guide.
  2. Refer to Getting Started with Performance Analysis in PyTorch Training Scenarios to modify the training script and start distributed training for data collection.

Data Display

The profile data in cluster scenarios needs to be displayed on the GUI of MindStudio Insight. For details, see MindStudio Insight User Guide.