Documentation

Command Syntax

Common Options

The command for accuracy comparison is structured as follows:

python3 msaccucmp.py compare -m my_dump_path -g golden_dump_path [-f fusion_rule_file] [-cf close_fusion_rule_file] [-q quant_fusion_rule_file] [-out output] [-map] [-c custom_script_path] [-alg algorithm] [-v version] [-r range] [-overflow_detection] [-max]

Table 1 describes the command-line options.

Table 1 Command-line options for accuracy comparison

Option

Description

Required

-m

--my_dump_path

Directory for storing the data file generated during model running on the Ascend AI Processor.

There are multiple dump data files in binary format. You need to specify their parent directory, for example, $HOME/MyApp_mind/resnet50. The resnet50 folder is used to store the dump data files.

Training scenario:

  • TensorFlow can be used as the original training network for comparison.
  • During single-file comparison, you need to specify the directory where the data file is located.
  • If batch comparison on multiple dump data files is supported, you can specify dump_path/time/ as a fixed path. Only TensorFlow can be used as the original training network for comparison. The fixed path can store multiple dump data files. Each dump data file must have a unique sub-path, which is named in the format of dump_path/time/device_id/model_name/model_id/dump_step/dump file.

Yes

-g

--golden_dump_path

Directory for storing the data file of the original network running on the GPU/CPU.

There are multiple .npy data files. You need to specify their parent directory, for example, $HOME/Standard_caffe/resnet50. The resnet50 folder is used to store the .npy data files.

If the -cf option is used, it specifies the directory of the dump data file generated after model conversion with operator fusion disabled.

Yes

-f

--fusion_rule_file

Network-wide information file.

Inference scenario:

  • A .json file converted from the .om model file using ATC, which contains the mapping between network-wide operators.
  • This option specifies the .json file generated during model conversion when operator fusion is enabled by default. To specify the .json file generated during model conversion when operator fusion is disabled, use the -cf option.

Training scenario:

  • It is a .json file converted from the .txt graph file using ATC.
  • For single-file comparison, set this option to a specific .json file. For batch comparison, set this option to a directory where multiple .json files are stored.

No

-cf

--close_fusion_rule_file

Network-wide information file (a .json file converted from the .om model file using ATC, which contains the mapping between network-wide operators when operator fusion is disabled).

For details about how to use this option, see Comparison Operation and Analysis.

This option is supported only in the inference scenario.

No

-q

--quant_fusion_rule_file

Quantization information file (.json output by Ascend model compression).

The quantization information file in .json format is generated through AMCT-based quantization. It contains the mapping between network-wide quantization operators and is used for operator matching during accuracy comparison.

In scenarios where comparison is performed between non-quantized original Caffe models and quantized offline models, either this option or the -f option must be specified. In scenarios where comparison is performed between non-quantized original Caffe models and quantized original models, only this option is used.

This option is supported only in the inference scenario.

No

-out

--output

Path of the comparison result. Defaults to the current path.

You are not advised to configure directories that are different from those of the current user to avoid privilege escalation risks.

Training scenario:

  • For single-file comparison, the result file is named in the result_{timestamp}.csv format.
  • For batch comparison, the result file is named in the {device_id}_{model_name}_{dump_step}_result_{timestamp}.csv format (multiple .csv result files are generated).

No

-map

--mapping

GPU/NPU mapping table output.

Generally, the GPU/NPU mapping table can be obtained from the .csv file only after accuracy comparison is complete. This parameter directly extracts the GPU/NPU mapping table before accuracy comparison. You are advised to use it when a large amount of data needs to be compared and the GPU/NPU mapping table needs to be obtained in advance. For details, see Obtaining the GPU/NPU Mapping Table.

No

-c

--custom_script_path

Path for storing customized script files, including algorithm .py files and format conversion .py files. Specify this path to the parent directory of the script directory. When specifying this option, check whether the following directories exist:

  • If the directory for storing customized algorithm .py files exists and its name is algorithm, the tool reads the algorithm .py files in the algorithm directory to generate a customized algorithm. You can use the -alg option to specify the customized algorithm as the comparison algorithm. For details about the requirements on customized algorithm .py files, see Preparing a Customized Algorithm .py File.
  • If the directory for storing customized format conversion .py files exists and its name is format_convert, the tool reads the format conversion .py files in the algorithm directory. For details about the requirements on customized format conversion .py files, see Preparing a Customized .py File for Format Conversion.

You are advised not to invoke customized script files in directories of other users to avoid privilege escalation risks.

No

-alg

--algorithm

Comparison algorithm dimension. Possible values are as follows:

  • 0: CosineSimilarity, which specifies the cosine similarity algorithm.
  • 1: MaxAbsoluteError, which specifies the maximum absolute error algorithm.
  • 2: AccumulatedRelativeError, which specifies the cumulative relative error algorithm.
  • 3: RelativeEuclideanDistance, which specifies the relative Euclidean distance algorithm.
  • 4: KullbackLeiblerDivergence, which specifies the Kullback-Leibler divergence algorithm.
  • 5: StandardDeviation, which specifies the standard deviation algorithm.
  • 6: MeanAbsoluteError, which specifies the mean absolute error.
  • 7: RootMeanSquareError, which specifies the root mean square error.
  • 8: MaxRelativeError, which specifies the max. relative error.
  • 9: MeanRelativeError, which specifies the mean relative error.
  • all (default): All algorithms are used for comparison (including customized and built-in algorithms).
  • Customized algorithm name (algorithm_name). The algorithm name is determined by the customized algorithm file. For example, if the customized algorithm file name is alg_RelativeError.py, the customized algorithm name is RelativeError. The -c option must be configured together.

You can select multiple values (numbers or algorithm names) and separate them by commas (,), for example, 0,1,2,RelativeError.

If only some dimensions are selected, only the selected dimensions are displayed in the comparison result.

No

-a

--algorithm_options

Advanced options of a specified comparison algorithm. The specified algorithm must be a built-in algorithm under the -alg option or a customized algorithm under the -c option. After these advanced options are set for the specified algorithm, it will be executed based on the option values.

Input format: Algorithm_name:Parameter_name=Value,Parameter_name=Value;Algorithm_name:Parameter_name=Value,Parameter_name=Value. Separate options with commas (,), and algorithms with semicolons (;), for example, CosineSimilarity:max=1,min=0;aa:max=1,min=0. The algorithm name must be the same as the value of algorithm_name in the customized algorithm .py file of the -c option. For details, see Preparing a Customized Algorithm .py File.

No

-v

--version

Dump file type. 1: data file after Protobuf serialization; 2 (default): data file in a custom format.

No

-r

--range

Operator comparison range. The configuration method is as follows:

  • start: first operator to be compared. The value range is [1, number of operators involved in computing]. The default value is 1.
  • end: last operator to be compared. The value can be -1 or [start, number of operators involved in computing]. The default value is -1, indicating that the last operator involved in computing in the network model is dynamically obtained.
  • step: the start + step x n operator to be compared. The value range of step is [1, Number of operators involved in computing). The default value is 1, and n is a positive integer starting from 1.

The format is "start,end,step". For example, -r 1,101,20 indicates that the tensors of operators 1, 21, 41, 61, 81, and 101 are compared.

If this option is not configured, all operators involved in computing in the network model are compared.

In scenarios where comparison is performed between non-quantized original Caffe models and quantized original models and between models running on two different Ascend devices, this option cannot be configured.

You need to configure the -f or -cf option first to specify the network-wide information file of the offline model.

You can configure either -s or -r.

No

-s

--select

Operator comparison range. Select the operators to be compared by specifying the operator index (operator IDs in the network model). The format is as follows: -s 1,2,3. Values are separated by commas (,).

You need to configure the -f or -cf option first to specify the network-wide information file of the offline model.

You can configure either -s or -r.

No

-max

--max_cmp_size

Maximum number of bytes for each dump data comparison. This option is used to speed up the accuracy comparison process. The default value is 0, indicating full comparison. The unit is byte. If the operator output of a model has a large shape, the comparison is time-consuming. In this case, specific configuration is required.

No

-overflow_detection

Network-wide operator overflow/underflow detection. Overflow/Underflow operators can be detected during network-wide comparison.

Network-wide operator overflow/underflow detection is disabled by default.

When the tensor data type of an operator is fp16 and any absolute value in the tensor is greater than or equal to 65504, the operator overflows.

In scenarios where comparison is performed between non-quantized original Caffe models and quantized original models, this option does not take effect.

No

Single-Operator Comparison

The command for single-operator comparison is structured as follows:

python3 msaccucmp.py compare -m my_dump_path -g golden_dump_path [-f fusion_rule_file] [-cf close_fusion_rule_file] [-q quant_fusion_rule_file] [-out output] [-op op_name] [-o output_tensor] [-i input_tensor] [-c custom_script_path] [-v version] [-n topn] [--ignore_single_op_result] [-ml max_line] [-overflow_detection]

Table 2 describes the command-line options.

Table 2 Command-line options for single-operator comparison

Option

Description

Mandatory (Yes/No)

-op

--op_name

Name of the single-operator to be compared. The operator name can be obtained from:

  • Inference scenario:
    • the .om model.
    • the .json file generated by converting the .om model file using ATC.
    • NPUDump field in the network-wide comparison result file.
  • Training scenario:
    • the training model.
    • Computational graph file (.txt).
    • NPUDump field in the network-wide comparison result file.

Yes

-o

--output_tensor

Index of the output data to compare.

The configuration format is -o Index, in which the Index can be obtained from the output of the TensorIndex field in the network-wide comparison result file. For example, if TensorIndex is trans_Cast_0:input:0, the Index is 0.

If neither -o nor -i is included, the output indexed 0 is compared.

Valid only when -op is configured. Mutually exclusive with -i.

No

-i

--input_tensor

Index of the input to compare.

The configuration format is -i Index, in which the Index can be obtained from the input of the TensorIndex field in the network-wide comparison result file. For example, if TensorIndex is trans_Cast_0:input:0, the Index is 0.

Valid only when -op is configured. Mutually exclusive with -o.

No

-n

--topn

Only the first n data items of the absolute error and relative error are displayed. After the comparison is complete, the comparison results are displayed on the screen and the .csv result files are generated. The value range is [1, 10000], and the default value is 20. This option is valid only when -op is configured.

The names of the generated .csv result files are as follows:

  • Absolute error: {op_name}_{input/output}_{index}_absolute_error_topn.csv
  • Relative error: {op_name}_{input/output}_{index}_relative_error_topn.csv

No

--ignore_single_op_result

Complete data of single-operator comparison is not generated in each .csv result file. That is, no .csv file with complete comparison data is generated. This option is valid only when -op is configured.

If this option is not set, complete comparison results are generated.

No

-ml

--max_line

Maximum number of records contained in a single .csv file generated during single-operator comparison. The value range is [10000,1000000], and the default value is 1000000.

If the number of records in the single-operator comparison result file is large, you can set this option to split the .csv file into multiple files. For example, if the number of data records is 100000 and this option is set to 10000, 10 .csv files are generated.

Valid only when -op is configured; not take effect when --ignore_single_op_result is configured.

No
Parent topic: Appendixes