Adding Dynamic Batch Size and Image Size Support to an Offline Model

Overview

In certain inference scenarios, for example, where an object recognition network is executed after object detection, the batch size of the network input is dynamic due to the unfixed number of objects. It would be a great waste of computing resources to perform each inference using the maximum batch size or image size.

To address this issue, ATC provides the --dynamic_batch_size option for setting the batch size profiles and the --dynamic_image_size option for setting the image size profiles.

Procedure

The following conversion example takes the TensorFlow ResNet50 network model for illustration. Click here to download the model file (*.pb) of the ResNet50 network.

1. Log in to the development environment as the CANN running user and upload the model file (.pb) used for model conversion to any directory in the development environment, for example, $HOME/module/.
2. Run the following command (the path and file arguments in the command are for reference only):
   • Dynamic batch size

     atc --model=$HOME/module/resnet50_tensorflow_1.7.pb  --framework=3 --output=$HOME/module/out/tf_resnet50 --soc_version=<soc_version> --input_shape="Placeholder:-1,224,224,3"  --dynamic_batch_size="1,2,4,8"  

   • Dynamic image size

     atc --model=$HOME/module/resnet50_tensorflow_1.7.pb  --framework=3 --output=$HOME/module/out/tf_resnet50 --soc_version=<soc_version>  --input_shape="Placeholder:1,-1,-1,3"  --dynamic_image_size="224,224;448,448"  

   The key options are described as follows:

   • --dynamic_batch_size: sets dynamic batch size profiles.
   • --dynamic_image_size: sets dynamic image size profiles.
   • --input_shape: specifies the shape of the model input data. This option is used together with --dynamic_batch_size or --dynamic_image_size.
   • --model: specifies the path of the ResNet50 model file.
   • --framework: specifies the original framework type. The value 3 indicates TensorFlow.
   For details about the options, see Command Line Options. If the following information is displayed, the model conversion is successful. If the model conversion fails, locate the fault by referring to "Error Codes".

   1	ATC run success, welcome to the next use.

   Find the generated offline model (for example, tf_resnet50.om) in the directory specified by the --output argument.

   In your OM offline model, you will find a newly added input (for example, Data in the red outline in Figure 1), which provides a specific batch size (or image size) for model inference. Assume that the batch size (or image size) of input a is dynamic. In the generated OM offline model, input b is added to describe the batch size (or image size) of input a.

   Figure 1 Offline model with dynamic batch sizes