Adding Dynamic Batch Size and Image Size Support to an Offline Model

Overview

In certain inference scenarios, for example, where an object recognition network is executed after object detection, the batch size of the network input is dynamic due to the unfixed number of objects. It would be a great waste of computing resources to perform inferences using the maximum batch size or image size. Therefore, when using ATC, you can apply --dynamic_batch_size or --dynamic_image_size to your model conversion to set the supported batch size or image size, respectively.

Procedure

The following conversion example uses a TensorFlow ResNet-50 network model for illustration. Refer to 1 to obtain the original model file.

1. Log in to the development environment as the CANN running user and upload the model file (.pb) used for model conversion to any directory in the development environment, for example, $HOME/module/.
2. Include --dynamic_batch_size or --dynamic_image_size in your atc command to generate an offline model (the path and file arguments in the command are for reference only).
   • Dynamic batch size

     atc --model=$HOME/module/resnet50_tensorflow*.pb  --framework=3 --output=$HOME/module/out/tf_resnet50 --soc_version=<soc_version> --input_shape="Placeholder:-1,224,224,3"  --dynamic_batch_size="1,2,4,8"  

   • Dynamic image size

     atc --model=$HOME/module/resnet50_tensorflow*.pb  --framework=3 --output=$HOME/module/out/tf_resnet50 --soc_version=<soc_version>  --input_shape="Placeholder:1,-1,-1,3"  --dynamic_image_size="224,224;448,448"  

   For details about the command-line options, see Command Line Options. You should see information similar to the following if the conversion is successful. If it fails, refer to Troubleshooting to locate the fault.

   1	ATC run success

   Find the generated offline model (for example, tf_resnet50.om) in the directory specified by the --output argument.

   In the result .om model, you will find a newly added input, which provides the runtime batch size (or image size) for model inference. Assume that the batch size (or image size) of input a is dynamic. In the generated OM model, input b is added to describe the batch size (or image size) of input a.