Comparison Results

1

Menu bar

From left to right, there are four functions: Open..., New Task, Refresh, and Help. Open... is used to open and display the comparison result .csv file. New Task is used to create a comparison task. Refresh is used to read and refresh files managed in File Manager. Help is used to view the restrictions and suggestions on using Model Accuracy Analyzer as well as the links to online courses.

2

File Manager, historical data management

This area displays the specified folder, network-wide comparison .csv files generated in the folder, and CSV files that are opened by clicking Open.... You can open, delete, and save historical data in the folder and CSV files, right-click a folder to delete it, and right-click in the blank area to create a comparison task. Refresh or open and display the comparison result files in CSV format.

3

Model Accuracy Analysis, accuracy comparison and analysis

Only operators with results are displayed by default. You can click a column name to sort the data and click Show Invalid Data to display the data that cannot be compared. For details about the fields in each column, see Table 2.

4

Scatter Diagram, scatter distribution chart of each algorithm metric.

Show Model, visualized display of comparison models

Scatter Diagram: The horizontal coordinate indicates the operator execution sequence, and the vertical coordinate indicates the actual value of the algorithm metric in the corresponding tensor. For the meaning of each field, see Table 3.

Show Model: Displays the NPU and Ground Truth model diagrams. For details, see Table 4.

Note: The historical data management function does not support data aging. That is, when the disk space is insufficient, the system cannot automatically delete unnecessary historical files. You need to delete unnecessary files manually to ensure the proper running of the comparison program.

Index

ID of an operator in a network model.

OpSequence

Sequence in which an operator runs. That is, ID of the operator in the network-wide information file. This parameter is available only when Operator Range is set.

OpType

Operator type.

NPUDump

Operator name of the NPU Dump model. When the cursor hovers over an operator, the file path of the operator is displayed.

DataType

Data type of operators on the NPU Dump side.

Address

Virtual memory address of the dump tensor, which detects memory faults of an operator. The address can be extracted only for network-wide comparison of dump data files generated during network running on Ascend AI ProcessorNPU IP Accelerator.

GroundTruth

Operator name of the Ground Truth model. When the cursor hovers over an operator, the file path of the operator is displayed.

DataType

Data type of operators on the Ground Truth side.

TensorIndex

Operator input ID and output ID of the NPU Dump model.

Shape

Shape of the compared tensor.

OverFlow

Overflow/Underflow operator. YES indicates that overflow/underflow occurs on an operator. NO indicates that no overflow/underflow occurs on the operator. NaN indicates that overflow/underflow detection is not performed.

This parameter is displayed when the Advisor function is enabled. It provides data for FP16 overflow/underflow detection expert suggestions in Expert Suggestions on Comparison Results.

CosineSimilarity

Result of the cosine similarity comparison. The value ranges from -1 to 1. A value closer to 1 indicates higher similarity.

MaxAbsoluteError

Result of the maximum absolute error comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

KullbackLeiblerDivergence

Result of the Kullback-Leibler divergence comparison. The value ranges from 0 to infinity. The smaller the Kullback-Leibler divergence, the closer the approximate distribution is to the true distribution.

RootMeanSquareError

Root mean square error. The value ranges from 0 to infinity. If values of both MeanAbsoluteError and RootMeanSquareError are close to 0, the measured value is more approximate to the actual value. If the value of MeanAbsoluteError is close to 0, a larger value of RootMeanSquareError indicates that some values are excessively large. A larger value of MeanAbsoluteError and RootMeanSquareError value equal to or approximate to that of MeanAbsoluteError indicate that the overall deviation is more centralized. A larger value of MeanAbsoluteError and RootMeanSquareError value larger than that of MeanAbsoluteError indicate that the overall deviation exists and its distribution is scattered. Other situations do not exist because "RMSE ≥ MAE" is always true.

MaxRelativeError

Max. relative error. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

RelativeEuclideanDistance

Result of the Euclidean relative distance comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

StandardDeviation

Result of the standard deviation comparison. The value ranges from 0 to infinity. The smaller the standard deviation is, the smaller the dispersion is, and the closer the value is to the average value. The mean value and standard deviation of the dump data are displayed in the format of (mean value;standard deviation). The first set of data is the result of NPU Dump, and the second set is the result of Ground Truth.

AccumulatedRelativeError

Result of the accumulated relative error comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

MeanAbsoluteError

Mean absolute error. The value ranges from 0 to infinity. If values of both MeanAbsoluteError and RootMeanSquareError are close to 0, the measured value is more approximate to the actual value. If the value of MeanAbsoluteError is close to 0, a larger value of RootMeanSquareError indicates that some values are excessively large. A larger value of MeanAbsoluteError and RootMeanSquareError value equal to or approximate to that of MeanAbsoluteError indicate that the overall deviation is more centralized. A larger value of MeanAbsoluteError and RootMeanSquareError value larger than that of MeanAbsoluteError indicate that the overall deviation exists and its distribution is scattered. Other situations do not exist because "RMSE ≥ MAE" is always true.

MeanRelativeError

Mean relative error. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

CompareFailReason

Cause of the comparison failure.

If the cosine similarity is 1, check whether the input or output shapes of the operator are empty or all 1. If the input or output shapes of the operator are empty or all 1, the input or output of the operator is a scalar. In this case, the following message is displayed: "this tensor is scalar."

Note 1: If the results of cosine similarity and Kullback-Leibler divergence are NaN, and the results of other algorithms exist, at least one piece data on the left or the right is 0. If the result of Kullback-Leibler divergence is inf, one piece data on the right is 0. If NaN is displayed, the dump data contains NaN.

Note 2: Hover the cursor over the table header to view the parameter description.

Note 3: If custom algorithm comparison is configured, add the custom algorithm column after the built-in algorithms in the comparison result.

Algorithm

Displays the scatter chart of the algorithm comparison result. StandardDeviation, KullbackLeiblerDivergence, and AccumulatedRelativeError are not supported.

Tensor

Filters and displays the input and output results in a scatter chart.

Highlight

Highlights the scatters of the operator tensors. Drag the slider between [min,max] of the corresponding algorithm metric to set the metric threshold (vertical coordinate). The point whose value is greater than or equal to the threshold is displayed in blue, and the point whose value is less than the threshold is displayed in red. For example, for cosine similarity, if the threshold is set to 0.98, the operator tensors whose values are less than 0.98 are displayed in red.

Note 1: When you move the cursor to a tensor point, the tensor information is displayed, including Index (index of the operator corresponding to the tensor), Op Name (operator name), Tensor Index (tensor type, input or output), and Value (tensor value in the current algorithm dimension).

Note 2: The scatter chart can be zoomed in or out.

Note 3: Tensors you specify are highlighted in area 3.

NPU Model

Offline model visualization. Specify the operator mapping file (.json) or offline model file (.om).

In the training scenario, if the model file used for network-wide comparison is a computational graph file (.txt), the model diagram cannot be displayed.

Ground Truth Model

Original model visualization. Specify the original model file.

Input Model

Specify the operator mapping file (.json), offline model file (.om), or original model file.

Note 3: If you specify tensors in area 3, the corresponding network model nodes are highlighted.