Comparison Operation and Analysis

Description

If the size of the specified data file for comparison exceeds 1 GB or the size of the .json file exceeds 100 MB, the comparison may take a long time and the system displays the message "The size (%d) of %s more than the XX, it needs more time to run."

Prerequisites

• Ensure that operations in Change History have been completed.
• Prepare the comparison file based on the scenario in Overview.

Procedure

1. Log in to the CANN tool installation environment.
2. Generate .json files.
   • Use the offline model with operator fusion enabled to generate the resnet50_on.json file.

     atc --mode=1 --om=$HOME/module/out/caffe_resnet50_on.om --json=$HOME/module/out/caffe_resnet50/resnet50_on.json

   • Use the offline model with operator fusion disabled to generate the resnet50_off.json file.

     atc --mode=1 --om=$HOME/module/out/caffe_resnet50_off.om --json=$HOME/module/out/caffe_resnet50/resnet50_off.json

   

   Skip this step in Accuracy Comparison Before and After Version Iteration scenarios.

3. Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann..
4. Run the comparison command.

   Use an offline model with operator fusion enabled and disabled to run on the Ascend AI Processor to generate corresponding dump data files.

   Save the data files in the following directories, respectively:

   • $HOME/MyApp_mind/resnet50_on
   • $HOME/MyApp_mind/resnet50_off

   In the Accuracy Comparison Before and After Version Iteration scenarios, the two data files to be compared are also saved in different directories.

   

   There are multiple dump and .npy data files used for comparison. Therefore, the -m and -g options in the following steps must specify the parent directory of the data files, for example, $HOME/MyApp/resnet50. The resnet50 folder is used to store the comparison data files.

   The directory structure is as follows:

   1 2 3 4 5 6

   root@xxx:$HOME/MyApp/resnet50# tree . ├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul_1.24.1614717261785536 ├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul.21.1614717261768864 ├── BatchMatMul.bert_encoder_layer_10_attention_self_MatMul_1.235.1614717263664916 # This is only an example. The remaining file names are omitted here.

   Specify a network-wide information file (by setting -f and -cf)

   python3 msaccucmp.py compare -m $HOME/MyApp_mind/resnet50_on -g $HOME/MyApp_mind/resnet50_off  -f $HOME/module/out/caffe_resnet50/resnet50_on.json -cf $HOME/module/out/caffe_resnet50/resnet50_off.json -out $HOME/MyApp_mind/out

   Or

   Do not specify a network-wide information file (by not setting -f and -cf)

   python3 msaccucmp.py compare -m $HOME/MyApp_mind/resnet50_on -g $HOME/MyApp_mind/resnet50_off -out $HOME/MyApp_mind/out

   

   The preceding command provides only examples of parameters required in the current scenario. For example, if the range of potential accuracy issues is known or the output file size of a large network model is too large, you can configure related parameters to reduce the output data volume. For details about more parameters, see Command Syntax.

   Table 1 Command-line options for network-wide comparison

   Option	Description	Mandatory (Yes/No)
   -m --my_dump_path	Directory for storing the data file of an offline model running on the Ascend AI Processor with operator fusion enabled.	Yes
   -g --golden_dump_path	Directory for storing the data file of an offline model running on the Ascend AI Processor with operator fusion disabled.	Yes
   -f --fusion_rule_file	Network-wide information file. .json file converted from an offline model (with operator fusion enabled) using ATC. You do not need to set this option in Accuracy Comparison Before and After Version Iteration scenarios.	No
   -cf --close_fusion_rule_file	Network-wide information file. .json file converted from an offline model (with operator fusion disabled) using ATC. You do not need to set this option in Accuracy Comparison Before and After Version Iteration scenarios.	No
   -out --output	Path of the comparison result. Defaults to the current path. You are not advised to configure directories that are different from those of the current user to avoid privilege escalation risks.	No

   Figure 1 shows the comparison result.

   Figure 1 Comparison result of the accuracy issues of fused operators
   

   Parameters in a Complete Model Comparison Result describes the fields in the comparison result.

   NPUDump indicates the name of the operator of the offline model for which operator fusion is enabled, and GroundTruth indicates the name of the operator of the offline model for which operator fusion is disabled.

5. Analyze the comparison result.

   For details, see Overview. If the comparison fails or exceptions occur (for example, NaN in the results), see Comparison Result Description.