Comparison Procedure

Description

The .json file and directory names in this section are only examples. Replace them with the actual ones. Ensure that the operation user has the read and write permissions on the result path specified by --out.
If you perform single-operator comparison on dump data generated by the same model (with fusion enabled/disabled) running on the Ascend AI Processor, you need to specify both or neither of the -f and -cf options.

Procedure

This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.

Generate .json files.

atc --mode=1 --om=$HOME/data/resnet50.om --json=$HOME/data/resnet50.json

Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest..

Perform single-operator comparison on the default configurations.

There are multiple dump and .npy data files used for comparison. Therefore, the -m and -g options in the following steps must specify the parent directory of the data files, for example, $HOME/MyApp/resnet50. The resnet50 folder is used to store the comparison data files.

The directory structure is as follows:

          
               root@xxx:$HOME/MyApp/resnet50# tree
.
├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul_1.24.1614717261785536
├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul.21.1614717261768864
├── BatchMatMul.bert_encoder_layer_10_attention_self_MatMul_1.235.1614717263664916
# This is only an example. The remaining file names are omitted here.

python3 msaccucmp.py compare -m $HOME/MyApp_mind/resnet50 -g $HOME/Standard_caffe/resnet50 -f $HOME/data/resnet50.json -out$HOME/result  -op pool5 -i 0

For details about the parameter settings, see Command Syntax.

Figure 1 and Figure 2 show the content of an accuracy comparison result file.

Figure 1 Single-operator comparison result (summary)

The single-operator comparison result summary is stored in {op_name}_input_{index}_summary.txt or {op_name}_output_{index}_summary.txt. The parameters are described as follows:

**Table 1** Single-operator comparison result summary parameters
Parameter	Description
TotalCount	Number of data records in the dump data of the operator.
NPUDump	Operator name of the My Output model.
GroundTruth	Operator name of the Ground Truth model.
Format	Data format.
MinAbsoluteError	Minimum absolute error.
MaxAbsoluteError	Maximum absolute error.
MinRelativeError	Minimum relative error.
MaxRelativeError	Maximum relative error.

Figure 2 Single-operator comparison result (complete)

The complete result of single-operator comparison is stored in {op_name}_input_{index}_{file_index}.csv or {op_name}_output_{index}_{file_index}.csv. Each file can record a maximum of one million data items. This result is not generated when the --ignore_single_op_result option is configured. Table 2 lists the parameters in the comparison result.

**Table 2** Parameters for the complete result of single-operator comparison
Parameter	Description
N C H W	Data coordinates.
NPUDump	Dump data of the My Output model operator.
GroundTruth	Dump data of the Ground Truth model operator.
RelativeError	Relative error. The value is obtained by dividing the AbsoluteError value by the dump value of the Ground Truth operator. If the dump value of the Ground Truth operator is 0, a hyphen (-) is displayed.
AbsoluteError	Absolute error. The value is the difference between the dump value of the My Output operator and that of the Ground Truth operator.

If the size of the output result file is too large, you can configure related parameters to reduce the output data volume. Select a proper comparison scenario based on Result Files and perform 6 to quickly identify the locations (inputs, outputs, and coordinate points) of operators with accuracy issues.

(Optional) Perform single-operator comparison to compare only top N data items.
```
python3 msaccucmp.py compare -m $HOME/MyApp_mind/resnet50 -g $HOME/Standard_caffe/resnet50 -f $HOME/data/resnet50.json -out $HOME/result  -op pool1 -o 0 -n 20
```
For details about the parameter settings, see Command Syntax.

Figure 3 shows the comparison result.

Figure 3 Comparison result of the first 20 data items

Figure 4 and Figure 5 show the content of the .csv result files generated after comparing the first 20 data items.

Figure 4 Absolute error (top 20)

Figure 5 Relative error (top 20)
Analyze the comparison result.
After the comparison result file is generated, sort the absolute or relative errors in descending order to find top N data items. You can assess whether the operator's accuracy meets the specified criteria based on your evaluation metrics. If the accuracy does not meet the specified criteria, submit the current evaluation data to the operator development personnel for internal logic analysis.

The closer the value of the absolute or relative error is to 0, the higher the accuracy is. You need to determine the actual accuracy that your operator needs to meet based on actual needs.

Parent topic: Single-Operator Comparison