Documentation

Network-wide Comparison

Command Syntax

The command for tensor comparison is structured as follows:

python3 compare_vector.py -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH [-f FUSION_JSON_FILE_PATH] [-q QUANT_FUSION_RULE_FILE_PATH] -o OUTPUT_PATH [-custom CUSTOM_PATH]
python3 compare_vector.py -l LEFT_DUMP_PATH -r RIGHT_DUMP_PATH -f FUSION_JSON_FILE_PATH -o OUTPUT_PATH [-custom CUSTOM_PATH]
  • -l: directory of the compared data of the My Output model
  • -r: directory of the compared data of the Ground Truth model
  • -o: path and file name of the comparison result

    You are not advised to configure directories that are different from those of the current user to avoid privilege escalation risks.

  • -f: (optional) network-wide information file (.json file generated by converting the .om model file or .json file generated by converting the .txt graph file using ATC).
  • -q: (optional) quantization fusion pattern file
  • -csv: writes the comparison result to a CSV file. This function is enabled when this parameter is set. By default, this function is disabled, indicating that the comparison result is saved as a common text file. This function is optional.
  • -custom: (optional) customized path to store the .py file for format conversion, which should be the parent directory of the format_convert directory. For details about the .py file requirements, see Preparing a Customized .py File for Format Conversion.

    You are advised not to invoke customized script files in directories of other users to avoid privilege escalation risks.

  • -ffts: enables the ffts+ mode for dump data. The value can be True (enabled) or False (disabled). The default value is False. This function is optional. Currently, this parameter is not supported in any scenario.

Select the -f or -q argument based on the data prepared in Comparison Data Description.

Comparison Procedure

To conduct tensor comparison, perform the following steps.

  • Replace the .json file and directory names in this example with the actual ones. The result and log paths must be created in advance and the running user must have the read and write permissions on the paths.
  • This section describes how to compare the dump data of a non-quantized model running on the Ascend AI Processor and the .npy file of a non-quantized Caffe model. The following parameters are based on this example. You can replace them as required.
  • To compare two groups of dump data generated based on the same model running on the Ascend AI Processor, ensure that the number of inputs and outputs, formats, as well as the shapes are the same. Otherwise, the comparison cannot be performed. In this scenario, only the -l, -r, and -o options are required.
  1. Log in to the development environment.
  2. Run the export command to set the environment variable and generate a .json file.
    1. Set the environment variable.

      export LD_LIBRARY_PATH=$HOME/Ascend/ascend-toolkit/latest/lib64:${LD_LIBRARY_PATH}

    2. Generate a .json file.

      /ascend-toolkit/latest/bin/atc --mode=1 --om=$HOME/data/resnet50.om --json=$HOME/data/resnet50.json

      /ascend-toolkit/latest/bin --mode=5 --om=ge_proto_00005_Build.txt --json=ge_proto_00005_Build.txt.json

  3. Go to the $HOME/Ascend/ascend-toolkit/latest/tools/operator_cmp/compare directory.
  4. Run the tensor comparison command as follows:

    python3 compare_vector.py -l $HOME/MyApp_mind/resnet50 -r $HOME/Standard_caffe/resnet50 -f $HOME/data/resnet50.json -o $HOME/result/result.txt

    python3 compare_vector.py -l $HOME/MyApp_mind/resnet50 -r $HOME/Standard_tf/resnet50 -f $HOME/data/ge_proto_00005_Build.txt.json -o $HOME/result/result.txt

    To save the file in .csv format, modify the command into -o $HOME/result/result.csv -csv.

  5. The tensor comparison result is saved to the result.txt file, as shown in Figure 1 and Figure 2.
    Figure 1 Comparison result (inference)
    Figure 2 Comparison result (training)

    The parameters are described as follows:

    • LeftOp: operator name of the My Output model.
    • RightOp: operator name of the Ground Truth model.
    • TensorIndex: operator input ID and output ID of the My Output model.
    • CosineSimilarity: result of the cosine similarity comparison. The value range is [–1, +1]. A value closer to 1 indicates higher similarity. Otherwise, it indicates greater difference.
    • MaxAbsoluteError: result of the maximum absolute error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
    • AccumulatedRelativeError: result of the accumulated relative error comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
    • RelativeEuclideanDistance: result of the relative Euclidean distance comparison. A value closer to 0 indicates higher similarity. Otherwise, it indicates greater difference.
    • KullbackLeiblerDivergence: result of the Kullback-Leibler divergence comparison. The value ranges from 0 to infinity. The smaller the Kullback-Leibler divergence, the closer the approximate distribution is to the true distribution.
    • StandardDeviation: result of the standard deviation comparison. The value ranges from 0 to infinity. The smaller the standard deviation is, the smaller the dispersion is, and the closer the value is to the average value. The mean value and standard deviation of the dump data are displayed in the format of (mean value;standard deviation). The first set of data is the result of My Output, and the second set is the result of Ground Truth.
    • The asterisk (*) indicates that a new operator on the NPU does not have a corresponding standard operator. NaN indicates that there is no comparison result.
    • If the results of cosine similarity and Kullback-Leibler divergence are NaN, and the results of other algorithms exist, at least one data on the left or the right is 0. If the result of Kullback-Leibler divergence is inf, one data on the right is 0. If NaN is displayed, the dump data contains NaN.
Parent topic: Tensor Comparison