[object Object][object Object]

The trend visualization function parses precision data collected by msProbe, identifies the module names at the model layer, and displays its location across iteration steps, ranks, and network model, helping you observe the precision data and analyze precision problems based on the overall data distribution.

Concepts

• msProbe: short for MindStudio Probe, is a precision debugging toolkit that can locate precision issues during model training or inference.
• dump: data collection function of MindStudio Probe. The collected data is called dump data.
• monitor: training status monitoring function of MindStudio Probe. The collected data is called monitor data.
• Three dimensions: dimensions of the data to be observed in the trend visualization tool, including step, rank, and module name (tensor target, distinguished by the network layer or operator name).

Usage Process

1. Install the tool and collect data. For details, see .
2. Use the command line tool to parse the precision data and generate an SQLite database file in .db format. For details, see .
3. Start the TensorBoard service and set the [object Object] parameter to the output path of the precision data parsing function.
4. Use a browser to open the TensorBoard service page and view the data in the [object Object] plugin window.

[object Object]

Environment Setup

Install msProbe by referring to .

If you choose to compile and install msProbe, you must configure [object Object] in the compile command to build the trend visualization plugin.

Data Preparation

• Dump data (collecting model data with [object Object] set to [object Object] or [object Object])
  • For details about how to collect data in the PyTorch framework, see .
  • For details about how to collect data in the MindSpore framework, see .
• monitor data (output [object Object] set to [object Object])
  • For details about the collection method, see .

Constraints

• The PyTorch and MindSpore frameworks are supported.

[object Object]

Function

Parse dump or monitor data, identifies module names at each model layer, determines the positions of module names in the step, rank, and network model based on the dump data flushing sequence, and save the parsing result to an SQLite database file in .db format.

Precautions

• dump data: Only the data collected at the [object Object] or [object Object] level (specified by [object Object]) is supported.
• monitor data: Only the data collected when the output [object Object] is set to [object Object] is supported.
• To effectively display the data trend, the flushed data range is [object Object]. Data beyond this range will be truncated. The [object Object] value will be converted to [object Object], and the [object Object] value will be converted to [object Object].
• There may be multiple parsed files. When you , the first [object Object] file passed to the directory is opened by default. Currently, file selection is not supported.

Syntax

Parameters

[object Object]undefined

Example

Parse data files in the [object Object] directory, automatically identify monitor data and dump data, and save the parsed SQLite database file in .db format to the [object Object] directory. The preceding operation is executed in a single process and counted by micro-step. Mapping is not used.

Output Description

After the dump data parsing command is executed successfully, the [object Object] file is generated in the [object Object] directory.

After the monitor data parsing command is executed successfully, the [object Object] file is generated in the [object Object] directory.

[object Object][object Object]

Trend analysis is to perform visualized analysis on the statistics of tensor objectives from three dimensions: steps, ranks, and module names. This helps you observe the precision data and analyze precision issues based on the overall trend distribution.

[object Object]

The following figure shows the trend visualization GUI, including area 1 (toolbar), area 2 (heat map), and area 3 (line chart).

• Area 1: toolbar, which allows you to select metrics, statistics, display dimensions, and dimension values, and provides label filtering and heat map mode settings.
• Area 2: heat map, which displays distribution of precision data across the other two dimensions based on the selected dimension's value.
• Area 3: line chart, which shows the trend of precision data for a selected point as its dimension value varies, triggered by clicking a point in the heat map under the chosen dimension value.

[object Object][object Object]

Server with Direct Connectivity

Pass [object Object] where the [object Object] file is generated to [object Object].

Logs are printed after TensorBoard is started.

In the preceding figure, [object Object] is the server address, and [object Object] is the port number. You can specify another port number using the [object Object] parameter.

[object Object]

Server Without Direct Connectivity

If the link cannot be opened (for example, the server cannot be directly connected and a VPN is required), try one of the following methods:

1. Manually set a proxy for the local computer network. For example, in Windows 10, add the server address (for example, [object Object]) in the manual proxy settings.

   

   Then, run the following command on the server:

   [object Object]

   Finally, enter [object Object] in the browser's address bar.

   [object Object]

2. Use Visual Studio Code to connect to the server and enter the following command in the Visual Studio Code terminal:

   [object Object]

   

   Press and hold [object Object] and click the link.

3. Transfer the image composition result file from the server to the local computer and install msProbe on the local computer to view the image composition result.

   Enter the following command on the PC:

   [object Object]

   Press and hold [object Object] and click the link.

[object Object]

Google Chrome is recommended. Perform the following operations to access the trend visualization page.

1. Enter the server address and port number in the address box of the browser and press [object Object], to access the TensorBoard page.
2. Click [object Object] in the upper left corner to access the trend visualization page.

[object Object]

By selecting the metric, statistic, display dimension, and dimension value on the toolbar, you can view a heat map showing how precision data is distributed across the other two dimensions under the selected dimension value. The following figure shows the page, and the following table describes the detailed operations.

[object Object]undefined

When the dimension is set to [object Object] and parallelism strategies are applied to a Megatron model, refer to to understand how the network layer data collected under each rank maps to the actual network-wide location.

[object Object][object Object]undefined[object Object]

By clicking a point in the heat map, you can view the trend of precision data for that point as its dimension value varies. The following figure shows the page, and the following table describes the detailed operations.

[object Object]undefined[object Object]

Function

This function visualizes the mapping between the network layer of each rank and the entire network in a .

Model parallelism in Megatron distributes a model across different ranks. As a result, the model layer data collected on each node may contain only a portion of the full model, and the location of those layers within the overall model is not immediately apparent. This function offers visualization support for multi‑node model parallelism, helping you rapidly identify how model layers are mapped to each device under the current parallelism configuration.

Precautions

Only the tensor parallelism, pipeline parallelism, virtual pipeline parallelism, and data parallelism are supported in the Megatron scenario. The number of ranks should be less than or equal to 1024 and the number of model layers should be less than or equal to 256. That is, [object Object] and [object Object] must be met.

Example

1. Create a Python script, for example, [object Object]. Copy the following code to [object Object] and modify the configuration under [object Object] as required.

   [object Object]

   For details about the parameters, see .

2. Run the following command to start the conversion.

   [object Object]

Output Description

After the [object Object] API is successfully called, a [object Object] file in the format of [object Object] is generated in the configured [object Object]. [object Object] is the size of the virtual pipeline parallel group calculated based on the input parameters such as [object Object].

View the PNG file.

The table below describes the PNG file.

[object Object]undefined[object Object][object Object]

Mapping configuration files provide input to the [object Object] parameter of the function. After the [object Object] parameter is configured, the parser sequentially replaces model layer or operator names in each precision data file with the keys and values specified in [object Object]. This is intended for scenarios requiring name simplification or cross-step name alignment.

The JSON file format and an example are provided below; keys and values are strings.

In the preceding format, the field on the left is the key (for example, [object Object]), and the field on the right is the value (for example, [object Object]). The preceding configuration indicates that [object Object] is replaced with [object Object], and [object Object] is replaced with [object Object].

[object Object][object Object]

Prototype

Parameters

Parameters need to be passed during instance initialization when a ParallelConfig instance is configured.

[object Object]undefined

Returns

None

[object Object]

1. How to use the trend visualization tool to compare the precision data files of two different experiments?

   The trend visualization tool does not distinguish between benchmark experiments and comparison experiments. It only compares the precision data files based on their input paths. To compare the precision data files of two different experiments, you need to manually move the subdirectories of the two groups of files to the same directory and then use the trend visualization tool to view and compare the files.

   Assume there are two dump data files: [object Object] and [object Object].

   [object Object]

   You can move the [object Object] and [object Object] subdirectories to the same directory by appending steps.

   [object Object]

   After the command is executed, the precision data of four steps can be obtained. In this case, comparing the precision data trends of step 0 and step 2 is equivalent to comparing the precision data trends of step 0 in the original [object Object] and [object Object].

   [object Object]

   You can also move the [object Object] and [object Object] subdirectories to the same directory by appending ranks.

   [object Object]

   After the command is executed, the precision data visualization results of four ranks are obtained. In this case, comparing the precision data trends of rank 0 and rank 2 in step 0 is equivalent to comparing the precision data trends of rank 0 in step 0 of the original [object Object] and [object Object].