Overview

Check the Comparison Result Description for this scenario.
This scenario applies only to comparison between the same processors.
NPU vs. NPU: Only accuracy comparisons between two non-quantized offline models and between two quantized offline models are supported.

This scenario includes the following sub-scenarios:

Pre- and Post-iteration accuracy comparison: Determines if accuracy issues exist following CANN software updates, model version iterations, or model optimizations by comparing two sets of accuracy data.
Accuracy comparison with and without operator fusion: By default, the ATC tool enables operator fusion during offline model conversion. To evaluate the accuracy difference caused by fusion, compare the data generated before and after fusion is enabled.

Accuracy Comparison Before and After Version Iteration

You need to check whether there is a decrease in accuracy of an offline model generated through ATC-based conversion due to the CANN version iteration, model version iteration, or model tuning when the model is running on the AI processor. The comparison is performed between two non-quantized models and between two quantized models. The required data files are listed as follows.

**Table 1** Data file requirements for comparison between non-quantized models
File	Description	How to Obtain
Dump data file of the non-quantized offline model running on the AI processor (before version iteration)	Benchmark data	In the offline inference scenario, the methods for obtaining the dump data of the NPU environment are the same for different frameworks. For details, see the following: Preparing Dump Data Files of an Offline Model
Dump data file of the non-quantized offline model running on the AI processor (after version iteration)	Data to be compared

**Table 2** Data file requirements for comparison between quantized models
File	Description	How to Obtain
Dump data file of the quantized offline model running on the AI processor (before version iteration)	Benchmark data	In the offline inference scenario, the methods for obtaining the dump data of the NPU environment are the same for different frameworks. For details, see the following: Preparing Dump Data Files of an Offline Model
Dump data file of the quantized offline model running on the AI processor (after version iteration)	Data to be compared

Accuracy Comparison Based on Model Conversion with Operator Fusion Enabled and Disabled

Generally, operator fusion is enabled by default when the ATC tool is used to convert an offline model. To evaluate the operator accuracy difference before and after fusion, you need to obtain the following:

Accuracy data dumped from the offline model converted using ATC with operator fusion enabled
Accuracy data dumped from the offline model converted using ATC with operator fusion disabled

Then compare the accuracy of the two datasets.

This scenario involves comparing a non-quantized model with fusion enabled with a non-quantized model with fusion disabled, and comparing a quantized model with fusion enabled with a quantized model with fusion disabled. The required data files are listed as follows.

**Table 3** Data file requirements for comparison between non-quantized models with operator fusion enabled and disabled
File	Description	How to Obtain
Non-quantized offline model file (.om) (with operator fusion disabled) Non-quantized offline model file (.om) (with operator fusion enabled)	Model files	Preparing Offline Model Files
Dump data file of the non-quantized offline model running on the AI processor (with operator fusion disabled)	Benchmark data	In the offline inference scenario, the methods for obtaining the dump data of the NPU environment are the same for different frameworks. For details, see the following: Preparing Dump Data Files of an Offline Model
Dump data file of the non-quantized offline model running on the AI processor (with operator fusion enabled)	Data to be compared

**Table 4** Data file requirements for comparison between quantized models with operator fusion enabled and disabled
File	Description	How to Obtain
Quantized offline model file (.om) (with operator fusion disabled) Quantized offline model file (.om) (with operator fusion enabled)	Model files	Preparing Offline Model Files
Dump data file of the quantized offline model running on the AI processor (with operator fusion disabled)	Benchmark data	In the offline inference scenario, the methods for obtaining the dump data of the NPU environment are the same for different frameworks. For details, see the following: Preparing Dump Data Files of an Offline Model
Dump data file of the quantized offline model running on the AI processor (with operator fusion enabled)	Data to be compared

Parent topic: Comparison Between NPU and NPU (Offline Inference)