Documentation

Comparison Operation and Analysis

Description

If the size of the specified data file for comparison exceeds 1 GB or the size of the .json file exceeds 100 MB, the comparison may take a long time and the system displays the message "The size (%d) of %s more than the XX, it needs more time to run."

Prerequisites

  • Ensure that operations in Before You Start have been completed.
  • Prepare the comparison file based on the scenario specified in Overview.

Procedure

  1. Log in to the CANN tool installation environment.
  2. Generate .json files.
    • Use the offline model with operator fusion enabled to generate the resnet50_on.json file. 
      atc --mode=1 --om=$HOME/module/out/caffe_resnet50_on.om --json=$HOME/module/out/caffe_resnet50/resnet50_on.json
    • Use the offline model with operator fusion disabled to generate the resnet50_off.json file. 
      atc --mode=1 --om=$HOME/module/out/caffe_resnet50_off.om --json=$HOME/module/out/caffe_resnet50/resnet50_off.json

    Skip this step in Accuracy Comparison Before and After Version Iteration and Accuracy Comparison in Scenarios with Inference Processor Switching of a Model scenarios.

  3. Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest..
  4. Run the comparison command.

    Use an offline model with operator fusion enabled and disabled to run on the Ascend AI Processor to generate corresponding dump data files.

    Save the data files in the following directories, respectively:

    • $HOME/MyApp_mind/resnet50_on
    • $HOME/MyApp_mind/resnet50_off

    There are multiple dump and .npy data files used for comparison. Therefore, the -m and -g options in the following steps must specify the parent directory of the data files, for example, $HOME/MyApp/resnet50. The resnet50 folder is used to store the comparison data files.

    The directory structure is as follows:

    1
2
3
4
5
6
    		 
    root@xxx:$HOME/MyApp/resnet50# tree
.
├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul_1.24.1614717261785536
├── BatchMatMul.bert_encoder_layer_0_attention_self_MatMul.21.1614717261768864
├── BatchMatMul.bert_encoder_layer_10_attention_self_MatMul_1.235.1614717263664916
# This is only an example. The remaining file names are omitted here.

    python3 msaccucmp.py compare -m $HOME/MyApp_mind/resnet50_on -g $HOME/MyApp_mind/resnet50_off  -f $HOME/module/out/caffe_resnet50/resnet50_on.json -cf $HOME/module/out/caffe_resnet50/resnet50_off.json -out $HOME/MyApp_mind/out

    The preceding command provides only examples of parameters required in the current scenario. For example, if the range of potential accuracy issues is known or the output file size of a large network model is too large, you can configure related parameters to reduce the output data volume. For details about more parameters, see Command Syntax.

    Table 1 Command-line options for network-wide comparison

    Option

    Description

    Mandatory (Yes/No)

    -m

    --my_dump_path

    Directory for storing the data file of an offline model running on the Ascend AI Processor with operator fusion enabled.

    Yes

    -g

    --golden_dump_path

    Directory for storing the data file of an offline model running on the Ascend AI Processor with operator fusion disabled.

    Yes

    -f

    --fusion_rule_file

    Network-wide information file. .json file converted from an offline model (with operator fusion enabled) using ATC.

    You do not need to set this option in Accuracy Comparison Before and After Version Iteration and Accuracy Comparison in Scenarios with Inference Processor Switching of a Model scenarios.

    No

    -cf

    --close_fusion_rule_file

    Network-wide information file.

    .json file converted from an offline model (with operator fusion disabled) using ATC.

    You do not need to set this option in Accuracy Comparison Before and After Version Iteration and Accuracy Comparison in Scenarios with Inference Processor Switching of a Model scenarios.

    No

    -out

    --output

    Path of the comparison result. Defaults to the current path.

    You are not advised to configure directories that are different from those of the current user to avoid privilege escalation risks.

    No
    Figure 1 shows the comparison result.
    Figure 1 Comparison result of the accuracy issues of fused operators

    Parameters in a Complete Model Comparison Result describes the fields in the comparison result.

    NPUDump indicates the name of the operator of the offline model for which operator fusion is enabled, and GroundTruth indicates the name of the operator of the offline model for which operator fusion is disabled.

  5. Analyze the comparison result.

    For details, see Comparison Result Analysis. If the comparison fails or exceptions occur (for example, NaN in the results), see Comparison Result Description.