Comparison Results

1

Menu bar

From left to right, there are four functions: Open..., New Task, Refresh, and Help. Open... is used to open and display the comparison result .csv file. New Task is used to create a comparison task. Refresh is used to read and refresh files managed in File Manager. Help is used to view the restrictions and suggestions on using Model Accuracy Analyzer as well as the links to online courses.

2

File Manager, historical data management

This area displays the specified folder, network-wide comparison .csv files generated in the folder, and CSV files that are opened by clicking Open.... You can open, delete, and save historical data in the folder and CSV files, right-click a folder to delete it, and right-click in the blank area to create a comparison task. Refresh or open and display the comparison result files in CSV format.

3

Model Accuracy Analysis, accuracy comparison and analysis

Only operators with results are displayed by default. You can click a column name to sort the data. For details about the fields in each column, see Table 2.

Select Show Invalid Data to display the data that cannot be compared. Select Show Highlight Data to display only the comparison result data that does not meet the highlight threshold. When Show Highlight Data is deselected, slide the Highlight slider on the right to mark the data that does not meet the highlight threshold in the table in red. Deselect Show Highlight Data to restore the table to the default status.

4

Scatter Diagram, scatter distribution chart of each algorithm metric.

Show Model, visualized display of comparison models

Scatter Diagram: The horizontal coordinate indicates the operator execution sequence, and the vertical coordinate indicates the actual value of the algorithm metric in the corresponding tensor. For the meaning of each field, see Table 3.

Show Model: Displays the NPU and Ground Truth model diagrams. For details, see Table 4.

Note: The historical data management function does not support data aging. That is, when the disk space is insufficient, the system cannot automatically delete unnecessary historical files. You need to delete unnecessary files manually to ensure the proper running of the comparison program.

Index

ID of an operator in a network model.

OpSequence

Sequence in which an operator runs. That is, ID of the operator in the network-wide information file. This parameter is available only when Operator Range is set.

OpType

Operator type.

NPUDump

Operator name of the NPU Dump model. When the cursor hovers over an operator, the file path of the operator is displayed.

DataType

Data type of operators on the NPU Dump side.

Address

Virtual memory address of the dump tensor, which detects memory faults of an operator. The address can be extracted only for network-wide comparison of dump data files generated during network running on Ascend AI Processor.

GroundTruth

Operator name of the Ground Truth model. When the cursor hovers over an operator, the file path of the operator is displayed.

DataType

Data type of operators on the Ground Truth side.

TensorIndex

Operator input ID and output ID of the NPU Dump model.

Shape

Shape of the compared tensor.

OverFlow

Overflow/Underflow operator. YES indicates that overflow/underflow occurs on an operator. NO indicates that no overflow/underflow occurs on the operator. NaN indicates that overflow/underflow detection is not performed.

This parameter is displayed when the Advisor function is enabled. It provides data for FP16 overflow/underflow detection expert suggestions in Expert Suggestions on Comparison Results.

CosineSimilarity

Result of the cosine similarity comparison. The value ranges from –1 to 1. A value closer to 1 indicates higher similarity.

RelativeEuclideanDistance

Result of the Euclidean relative distance comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

MaxAbsoluteError

Result of the maximum absolute error comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

MeanAbsoluteError

Mean absolute error (MAE). The value ranges from 0 to infinity.

• Values of MeanAbsoluteError and RootMeanSquareError that are closer to 0 indicate that the measured value is more accurate and closer to the actual value.
• If the value of MeanAbsoluteError is close to 0, a larger value of RootMeanSquareError indicates that some values are excessively large.
• A larger MeanAbsoluteError value and a RootMeanSquareError value that is equal to or close to the MeanAbsoluteError value suggest that the overall deviation is more centralized.
• A larger MeanAbsoluteError and a RootMeanSquareError value greater than that of MeanAbsoluteError indicate the presence of overall deviation and a scattered distribution of the deviation.
• Other situations do not exist because "RootMeanSquareError ≥ MeanAbsoluteError" is always true.

RootMeanSquareError

Root mean squared error (RMSE). The value ranges from 0 to infinity.

• Values of MeanAbsoluteError and RootMeanSquareError that are closer to 0 indicate that the measured value is more accurate and closer to the actual value.
• If the value of MeanAbsoluteError is close to 0, a larger value of RootMeanSquareError indicates that some values are excessively large.
• A larger MeanAbsoluteError value and a RootMeanSquareError value that is equal to or close to the MeanAbsoluteError value suggest that the overall deviation is more centralized.
• A larger MeanAbsoluteError and a RootMeanSquareError value greater than that of MeanAbsoluteError indicate the presence of overall deviation and a scattered distribution of the deviation.
• Other situations do not exist because "RootMeanSquareError ≥ MeanAbsoluteError" is always true.

MaxRelativeError

Max. relative error. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

MeanRelativeError

Mean relative error. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

AccumulatedRelativeError

Result of the accumulated relative error comparison. The value ranges from 0 to infinity. A value closer to 0 indicates a higher similarity.

StandardDeviation

Result of the standard deviation comparison. The value ranges from 0 to infinity. The smaller the standard deviation is, the smaller the dispersion is, and the closer the value is to the average value. The mean value and standard deviation of the dump data are displayed in the format of (mean value;standard deviation). The first set of data is the result of NPU Dump, and the second set is the result of Ground Truth.

KullbackLeiblerDivergence

Result of the Kullback-Leibler divergence comparison. The value ranges from 0 to infinity. The smaller the Kullback-Leibler divergence, the closer the approximate distribution is to the true distribution.

CompareFailReason

Cause of the comparison failure.

If the cosine similarity is 1, check whether the input or output shapes of the operator are empty or all 1. If the input or output shapes of the operator are empty or all 1, the input or output of the operator is a scalar. In this case, the following message is displayed: "this tensor is scalar."

Note 1: If the results of cosine similarity and Kullback-Leibler divergence are NaN, and the results of other algorithms exist, at least one piece data on the left or the right is 0. If the result of Kullback-Leibler divergence is inf, one piece data on the right is 0. If NaN is displayed, the dump data contains NaN.

Note 2: Hover the cursor over the table header to view the parameter description.

Note 3: If custom algorithm comparison is configured, add the custom algorithm column after the built-in algorithms in the comparison result.

Note 4: Clicking any result cell in the table will jump to the corresponding operator and highlight it on the right scatter distribution chart or model diagram; clicking any operator in the model under Show Model on the right will highlight the corresponding cell in the table on the left.

Algorithm

Displays the scatter distribution chart of the algorithm comparison result. StandardDeviation, KullbackLeiblerDivergence, and AccumulatedRelativeError are not supported.

Tensor

Filters and displays the input and output results in a scatter distribution chart.

Highlight

Operator accuracy threshold. By sliding the slider between [min, max] of an algorithm metric, you set the metric threshold (y-axis). Points that meet the threshold are displayed in blue, while points that do not meet the threshold are displayed in red. At the same time, the data that does not meet the threshold is displayed in red in the table on the left. For the CosineSimilarity algorithm, data below the threshold is considered not meeting the threshold, while for other algorithms, data above the threshold is considered not meeting the threshold.

Note 1: When you move the cursor to a tensor point, the tensor information is displayed, including Index (index of the operator corresponding to the tensor), Op Name (operator name), Tensor Index (tensor type, input or output), and Value (tensor value in the current algorithm dimension).

Note 2: The scatter distribution chart can be zoomed in or out.

Note 3: Tensors you specify are highlighted in area 3.

NPU Model

Offline model visualization. Specify the operator mapping file (.json) or offline model file (.om).

In the training scenario, if the model file used for network-wide comparison is a computational graph file (.txt), the model diagram cannot be displayed.

Ground Truth Model

Original model visualization. Specify the original model file.

Input Model

Specify the operator mapping file (.json), offline model file (.om), or original model file.

Note 1: When a tensor in area 3 is selected, the model network is redirected to the corresponding node and highlights it.

Note 2: When sliding the Highlight slider on the scatter distribution chart, nodes in the model network that do not meet the highlight threshold will be marked in red.