Comparing Dump Data

Data Preparation

Upload the precision_tool and precision_data (including the GPU/CPU benchmark data and NPU precision data) folders to any directory in the Toolkit installation environment. The following is an example of the directory structure:

├── precision_tool
│    ├── cli.py
│    ├── ...
├── precision_data
│    ├── npu
│    │    ├── debug_0  // Stores NPU dump data.
│    ├── tf
│    │    ├── dump     // Stores GPU/CPU dump data.
│    ├── ...

Dependency Installation

# graphviz is an optional dependency and needs to be installed only when operator subgraphs need to be drawn.
pip3 install rich graphviz
# ubuntu/Debian
sudo apt-get install graphviz
# fedora/CentOS
sudo yum install graphviz
Modify the config.py file in the precision_tool/lib/config directory.
# The atc and msaccucmp.py tools in the Toolkit package are required. Set this parameter to the Toolkit installation directory.
# By default, the Toolkit package is installed in /usr/local/Ascend. You do not need to change the value. If you specify a directory for installation, you need to change the value.
CMD_ROOT_PATH = '/usr/local/Ascend'

Data Comparison

1. Start the PrecisionTool interactive command line.

   python3 ./precision_tool/cli.py

2. The interactive command line interface is displayed. (To exit, press Ctrl+C.)

   PrecisionTool >

3. Run the ac -l [limit_num] (-c) command to compare the accuracy of the entire network. For details, see "precision_tool Command Reference" in TensorFlow 1.15 Model Porting Guide.

   PrecisionTool > ac -c

   The comparison time varies according to the data volume.

   The comparison result is saved in the precision_data/temp/vector_compare directory in CSV format.

Accuracy Analysis

Open the CSV file in the precision_data/temp/vector_compare directory and search for the first operator whose cosine similarity is less than 0.98 from top to bottom. For details, see "Network Accuracy Comparison Result File" in TensorFlow 1.15 Model Porting Guide. Alternatively, run the vcs -f [file_name] -c [cos_sim_threshold] -l [limit] command (see "precision_tool Command Reference" in TensorFlow 1.15 Model Porting Guide) to filter the comparison result. By default, the vcs command filters the result whose cosine similarity is less than 0.98 and finds the first operator. If no result is obtained when the threshold is 0.98, increase the threshold and try again.

The following figure shows an example of the vcs command output.

• Left: name of the dump operator running on the NPU.
• Right: name of the dump operator running on the GPU/CPU.
• Input and Output: cosine similarity comparison result of the operator inputs and outputs. The value range is [–1, +1]. A value closer to 1 indicates a higher similarity, and a value closer to –1 indicates a higher difference.

As shown in the preceding figure, the operator inputs are basically the same, but the first output is obviously different from the benchmark (the cosine similarity is 0.806927, which is less than 0.98), indicating that the operator may have a precision problem. If the operator inputs are obviously different, you need to find the comparison result of the input node.