Exporting and Comparing Subnetworks

Overview

The model inference result on the Ascend AI Processor may be different from that of the counterpart original model. The Model Accuracy Analyzer can locate the accuracy drop causes at layers that are not customized (which constitute a suspected problematic subnetwork) so that you can compare the inference results of these layers. To simplify the workflow, you can leverage the Caffe or TensorFlow subnetwork export capability provided by MindStudio to easily export the subnetwork. Figure 1 describes the export workflow.

To use the subnetwork export function of the TensorFlow AI framework, install TensorFlow 1.15 and Python 3 (3.7.5 to 3.7.11).
To use the subnetwork export function of the Caffe AI framework, install Caffe and ensure that it can run properly in Python 3.7 or Python 3.9. YOLOv2, YOLOv3, and quantized network models cannot be exported from Caffe subnetworks. For details about the OSs supported by Caffe, see the "Installation" module on the Caffe official website. Windows does not support the subnetwork export function of the Caffe framework.

Figure 1 Exporting a subnetwork

Convert the original model into an offline model adapted to Ascend AI Processors using ATC. For details, see Model Conversion and Tuning.
Start inference with the offline model.
If the inference result is inconsistent with that of the original model, use the Model Accuracy Analyzer to locate the problem layers, and export a subnetwork consisting of the problem layers of the original model by referring to Export Procedure. Repeat the subnetwork export and accuracy comparison workload to narrow down the problem layers.
Tune the located minimal subnetwork.
If the fault is rectified, convert the model again and perform model inference.

Export Portals

On the MindStudio page, choose Ascend > Dump Network. The subnetwork export page is displayed.

Export Procedure

On the MindStudio menu bar, choose Ascend > Dump Network. The subnetwork export page is displayed, as shown in Figure 2.

Figure 2 Exporting a subnetwork

**Table 1** Parameter and button description
Parameter and Button	Description
Dump Data Path	Dump data of the offline model configured in NPU Dump on the Model Accuracy Analyzer page. Click on the right to select the dump directory.
Offline Model File	Offline model file configured in Model File on the Model Accuracy Analyzer page. Click on the right to select the offline model file.
Deploy Model	Original network model file. Click on the right to select the corresponding file on the MindStudio installation server. For a model file that is compressed by using AMCT, select the quantized model file that can be deployed on Ascend AI Processor.
Deploy Weight	Weight file of the original network model. If the model file and weight file are stored in the same directory on the MindStudio installation server, this parameter is automatically set after you select the model file. Otherwise, click on the right to select it manually. For a weight file that is compressed by using AMCT, select the quantized weight file that can be deployed on Ascend AI Processor. NOTE: This parameter is available only when the original model is a Caffe model. Other frameworks do not require .caffemodel weight files.
Dump Output Path	Subnetwork export path. Click on the right to select an export directory. If the directory does not exist, create one.
Show Result	Displays the parsing result of the original model in a chart on the right.
Dump	Exports a subnetwork.

After setting the parameters, click Show Result to parse the original model file.
If Model Name of the original model file after conversion is different from that of the original Caffe model file (.prototxt) or TensorFlow model file (.pb), the dialog box shown in Figure 3 is displayed during parsing.
- Click Yes to continue the parsing and obtain the parsing results, as shown in Figure 4.
- Click No to return to the subnetwork export window.
Figure 3 Inconsistent Model Names

Figure 4 shows that the original model is successfully parsed.
Figure 4 Parsing results
Where,
- View the operator details.
  In the lower part of the central area in Figure 4, you can click on the left of a parameter to expand its details, and click to collapse the details of all operators.
- View the operator output shape.
  Figure 4 also displays the output format (for example, NCHW) and shape (for example, 1,3,224,224) of each operator on the connection lines.
- Search for an operator.
  The search area is in the lower middle part of the window shown in Figure 4.
  
  All operators on the network are listed in the search area. Enter an operator name in the search text box. The matched operators are displayed. Select an operator, and the operator is outlined green in the network topology on the left. Details about the operator are displayed in the central area, including the operator name and the inputs and outputs.
  
  Figure 5 Finding operator details
- Find operator details.
  Enter the information you want to query in the text box, as shown in Figure 4, and click to search one item at a time.
Select the first node (input node) of the suspected problematic subnetwork model, and then right-click and choose start. Select the last node (output node), and then right-click and choose end. The nodes between start and end will be highlighted in blue in this case.
- With the input and output nodes selected, if you want to modify the selection, right-click a new node and select the corresponding attribute as required. The color of the new node changes accordingly, and the previously selected node is restored to the original color.
- You cannot select start for a node and end for its preceding node.
Click Dump to export the subnetwork.
MindStudio searches for the subnetwork from the original network model in the background and export it. The export progress is displayed in the lower right corner. You can click the progress bar to view the detailed progress.

The message shown in Figure 6 indicates an export success.
Figure 6 Export success message

Viewing the Export Result

In the directory set in Dump Output Path on the MindStudio installation server, find the folder formatted as dump_result_xxxx_xx_xx_xx_xx_xx, where xxxx_xx_xx_xx_xx_xx is a timestamp. The folder contains:
- dump.prototxt: model file of the Caffe subnetwork
- dump.pb: model file of the TensorFlow subnetwork
- dump.caffemodel: weight file of the Caffe subnetwork
- dump_data: dump data of all operators between Start Layer and End Layer of the subnetwork
  This folder also contains the input data of the Start Layer, which is used as the input data for subnetwork inference.

Find the export log messages in the Output window in MindStudio. The following gives some example log messages.

// Define the input information of the subnetwork.
2020-05-22 16:27:57  Add Layer Into Prototxt:
name: "pool1_input_0"
type: "Input"
top: "conv1"
input_param {
  shape {
    dim: 1
    dim: 64
    dim: 112
    dim: 112
  }
}

// Export dump data of all operators of the subnetwork.
2020-05-22 16:27:57  Output Dump Path:/home/username/dumpdate/dumpresult/ir6/dump_result_2020_05_22_16_27_44/dump_data

// Dump data of a layer between the start layer and end layer in the subnetwork. Output File indicates the dump data of the Dump Layer output of the offline model. Input File indicates the data input of the Dump Layer input of the offline model.
2020-05-22 16:27:58  Dump Layer:[res2a_branch2c, bn2a_branch2c, scale2a_branch2c]
Output File:Conv2D.res2a_branch2cres2ares2a_relu.151.1585884515795667
Input File:Conv2D.res2a_branch2bres2a_branch2b_relu.149.1585884515771632
...

// Dump data of all operators between the Start Layer and End Layer.
2020-05-22 16:27:58  Export Layer:pool1,res2a_branch1,bn2a_branch1,scale2a_branch1,res2a_branch2a,bn2a_branch2a,scale2a_branch2a,res2a_branch2a_relu,res2a_branch2b,bn2a_branch2b,scale2a_branch2b,res2a_branch2b_relu,res2a_branch2c,bn2a_branch2c,scale2a_branch2c,res2a,res2a_relu

You can perform further accuracy comparison using the exported subnetwork. If the problem is solved, go to the next step to export the subnetwork and perform inference again.

Exporting the Subnetwork and Performing Inference Again

Convert the subnetwork exported in 1 into an offline .om file. For details, see Model Conversion and Tuning.
If the exported subnetwork contains no input data node, turn the AIPP switch off during model conversion.
Convert the input data of the start layer on the subnetwork exported in 1 into the binary format, and use it as the input of the offline model during inference.
1. Search for the input data of the start layer.
  The input data is the Input File of operators at the start layer, which is the Dump Layer during subnetwork export. Based on this information, search for the file that is named the same as the Input File in the dump_data directory on the MindStudio installation server.
2. Convert the input data in Protobuf format into the NumPy format.
  Log in to the MindStudio installation server, switch to the Ascend-CANN-Toolkit_installation_directory/ascend-toolkit/latest/tools/operator_cmp/compare directory, and run the following command for format conversion:
```
python3 shape_conversion.py -i<absolute path of the input data in Protobuf format> -format NCHW -o <path of the .npy file>
```
  You can replace the paths as required. After conversion, you can find the input data (.npy) in the specified path.
3. Convert the input data in NumPy format into the binary format.
  Replace the data path in the following script with the actual path and save the script as a .py file (for example, submodeldataprocess.py).
```
import numpy as np

# Convert the file in NumPy format into the binary format.
conv1_relu_0 = np.load(
    "/home/username/dumpdate/subModelData/xxx.npy")  # absolute path of the input data in NumPy format
conv1_relu_0 = conv1_relu_0.astype(np.float16)
conv1_relu_0.tofile(
    "/home/username/dumpdate/subModelData/xxx.bin")  # storage path of data in binary format
```
  Upload the submodeldataprocess.py script to any directory on the MindStudio installation server. Run the following command to convert the .npy file into the binary format:
```
python3 submodeldataprocess.py
```
  In the binary file storage path, you can find the converted .bin file.
Perform inference in the original environment using the exported subnetwork, and perform inference in Ascend AI Processor using the offline model and input data generated in 1 and 2. Compare the inference results. If the difference persists, refer to Accuracy Comparison to perform accuracy comparison using the subnetwork generated in 1 and the offline model generated in 1.
Repeat subnetwork export and accuracy comparison to narrow down the problem layers, and tune the located minimal subnetwork.

Parent topic: Extended Functions