Model Accuracy Analyzer Deployment

Overview

To facilitate accuracy tuning in training, the one-click accuracy analyzer provides common functions for accuracy analysis of training networks, including:

• Floating-Point Exception Detection
• Fusion Exception Detection
• Network-wide Accuracy Comparison
• Random Error Detection

This tool encapsulates the run parameters of TF Adapter and extends functions of the model accuracy analyzer (msaccucmp.py) in the CANN package, facilitating accuracy fault location.

Restrictions

• Overflow/Underflow data collection is mutually exclusive with accuracy data dump.
• Set a proper epoch number, which helps avert running out of disk space caused by a large number of files generated during overflow/underflow data collection or accuracy data dump.

One-Click Accuracy Analyzer Deployment

This tool is installation-free. Download the precision_tool directory from https://gitee.com/ascend/tools and upload it to the training directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

├── resnet                              // Training working directory.
│    ├── __init__.py     
│    ├── imagenet_main.py              // Script for training the network based on the ImageNet dataset.
│    ├── imagenet_preprocessing.py     // ImageNet preprocessing module.
 │    ├── resnet_model.py               // ResNet model file.
│    ├── resnet_run_loop.py            // Data input processing and run loop (for training, validation, and test).
│    ├── cifar10_main                  // Training entry point file.
│    ├── ...
│    ├── precision_tool           // Directory of the one-click accuracy analyzer
│    │    ├── cli.py                   
│    │    ├── ...

• If the CANN development and operating environments are set up on the same server, simply upload the precision_tool directory to the training directory.
• If the CANN development and operating environments are set up on separate servers, upload the precision_tool directory to the training directory in the CANN operating environment and any directory in the CANN development environment.
  

  • The CANN operating environment (where NPU training is run on the Ascend AI Processor) is mainly used for collecting accuracy data during training accuracy tuning.
  • The CANN development environment is mainly used for accuracy analysis during training accuracy tuning.

Typical Workflow

Figure 1 shows the workflow of using the one-click accuracy analyzer precision_tool to analyze accuracy when the CANN development and operating environments are deployed on the same server.

Figure 1 CANN development and operating environments set up on the same server

Figure 2 shows the workflow of using the one-click accuracy analyzer precision_tool to analyze accuracy when the CANN development and operating environments are deployed separately.

Figure 2 CANN development and operating environments set up separately