文档

Single-Operator Comparison

Command Syntax

The command for tensor comparison is structured as follows:

python3 compare_vector.py -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH [-f FUSION_JSON_FILE_PATH] [-q QUANT_FUSION_RULE_FILE_PATH] -o OUTPUT_PATH -d OP_NAME [-t DETAIL_TYPE] [-i DETAIL_INDEX] [-custom CUSTOM_PATH]
python3 compare_vector.py -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH -f FUSION_JSON_FILE_PATH -o OUTPUT_PATH -d OP_NAME [-t DETAIL_TYPE] [-i OUTPUT_INDEX] [-custom CUSTOM_PATH]

The command-line options are described as follows:

  • -l: directory of the compared data of the My Output model
  • -r: directory of the compared data of the Ground Truth model
  • -o: path of the comparison result

    You are not advised to configure directories that are different from those of the current user to avoid privilege escalation risks.

  • -f: (optional) network-wide information file of the offline model (.json file generated by converting the .om model file or .json file generated by converting the .txt graph file using ATC).
  • -q: (optional) quantization fusion pattern file
  • -d: name of the single-operator layer to be compared of the My Output model
  • -t: (optional) indicates whether the dump file is the input or output (default).
  • -i: (optional) input_index or output_index of the operator of the My Output model
  • -csv: writes the comparison result to a CSV file. If this parameter is set, the function is enabled. By default, this function is disabled, indicating that the comparison result is saved to a TXT file.
  • -custom: (optional) customized path to store the .py file for format conversion, which should be the parent directory of the format_convert directory. For details about the .py file requirements, see Preparing a Customized .py File for Format Conversion.

    You are advised not to invoke customized script files in directories of other users to avoid privilege escalation risks.

  • -ffts: enables the ffts+ mode for dump data. The value can be True (enabled) or False (disabled). The default value is False. This function is optional. Currently, this parameter is not supported in any scenario.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct tensor comparison, perform the following steps.

  • Replace the .json file and directory names in this example with the actual ones. The result and log paths must be created in advance and the running user must have the read and write permissions on the paths.
  • This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
  • Single-operator comparison between two groups of dump data generated based on the same model running on the Ascend AI Processor is not supported.
  1. Log in to the development environment.
  2. Run the export command to set the environment variable and generate a .json file.

    Set the following environment variable:

    export LD_LIBRARY_PATH=$HOME/Ascend/ascend-toolkit/latest/lib64:${LD_LIBRARY_PATH}

    Generate the .json file:

    /ascend-toolkit/latest/bin/atc --mode=1 --om=$HOME/data/resnet50.om --json=$HOME/data/resnet50.json

    /ascend-toolkit/latest/bin --mode=5 --om=ge_proto_00005_Build.txt --json=ge_proto_00005_Build.txt.json

  3. Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest..
  4. Run the tensor comparison command as follows:

    python3 compare_vector.py -l $HOME/MyApp_mind/resnet50 -r $HOME/Standard_caffe/resnet50 -f $HOME/data/resnet50.json -o $HOME/result -d pool5 -i 0

    python3 compare_vector.py -l $HOME/MyApp_mind/resnet50 -r $HOME/Standard_tf/resnet50 -f $HOME/data/ge_proto_00005_Build.txt.json -o $HOME/result -d gradients/AddN_63 -i 0

  5. Figure 1 and Figure 2 show the content of a tensor comparison result file.
    Figure 1 Single-operator comparison result (summary)

    The single-operator comparison result summary is stored in {op_name}_input_{index}_summary.txt or {op_name}_output_{index}_summary.txt. The parameters are described as follows:

    • TotalCount: number of data records in the dump data of the operator
    • LeftOp: operator name of the My Output model
    • RightOp: operator name of the Ground Truth model
    • Format: data format
    • MinAbsoluteError & MaxAbsoluteError: minimum and maximum absolute error ranges
    • MinRelativeError & MaxRelativeError: minimum and maximum relative error ranges
    Figure 2 Single-operator comparison result (complete)

    The complete result of single-operator comparison is stored in {op_name}_input_{index}_{file_index}.csv or {op_name}_output_{index}_{file_index}.csv. Each file can record a maximum of one million data items. The parameters in Figure 2 are described as follows:

    • N C H W: data coordinates
    • Left: dump data of the My Output model operator
    • Right: dump data of the Ground Truth model operator
    • RelativeError: relative error. The value is obtained by dividing the AbsoluteError value by the dump value of the Ground Truth operator. If the dump value of the Ground Truth operator is 0, a hyphen (-) is displayed.
    • AbsoluteError: absolute error. The value is the difference between the dump value of the My Output operator and that of the Ground Truth operator.
Parent topic: Tensor Comparison