--dynamic_batch_size

Applicability

Description

Sets dynamic batch size profiles, which applies to the scenario where the number of images or sentences processed per inference batch is unfixed.

In certain inference scenarios, for example, where an object recognition network is executed after object detection, the batch size of the network input is dynamic due to the unfixed number of objects. It would be a great waste of computing resources to perform each inference using the maximum batch size or image size. Thanks to ATC's support for dynamic batch sizes and dynamic image sizes, you can include this option and the --dynamic_image_size option in your model conversion command to set the supported batch size or image size profiles, respectively.

In the OM model generated after conversion, you will find a newly added input, which provides the specific batch size for model inference. Assume that the batch size of input a is dynamic. In the generated OM model, input b is created to describe the specific batch size of input a.

See Also

Use this option together with --input_shape. This option is mutually exclusive with --dynamic_image_size, --input_shape_range, and--dynamic_dims. Only the scenario where N is in the first place of the shape is supported. That is, the first place of the shape is set to -1. If N is not in the first place, use --dynamic_dims to set it.

Argument

Argument: Batch size profiles, for example, "1,2,4,8".

Format: Enclose the whole argument in double quotation marks (""), and separate the batch sizes with commas (,).

Restrictions:

• For the following products, the batch size profile range is (1,100]. That is, at least two profiles must be set and a maximum of 100 profiles are supported. The recommended value range for each profile is [1, 2048].

  Atlas A3 training products / Atlas A3 inference products

  Atlas A2 training products / Atlas A2 inference products

  Atlas 200I/500 A2 inference products

  Atlas inference products

  Atlas training products

• In the scenario where you have set too large batch sizes or too many batch size profiles, you are advised to run the swapoff -a command to disable the use of swap space as memory to prevent slow operating environment.

Suggestions and Benefits

• Too large batch sizes or too many batch size profiles will cause model conversion failures.
• For computer vision (CV) networks, the recommended value of --dynamic_batch_size is 8 or 16. In this scenario, the network performance is better than that when the batch size is 1.
• For optical character recognition (OCR) or natural language processing (NLP) networks, the recommended value of --dynamic_batch_size is an integer multiple of 16.

Example

--input_shape="data:-1,3,416,416;img_info:-1,4"  --dynamic_batch_size="1,2,4,8"

-1 in --input_shape indicates that the dynamic batch size is set. The input supports the following shape profiles at ATC build time:

Profile 0: data(1,3,416,416)+img_info(1,4)

Profile 1: data(2,3,416,416)+img_info(2,4)

Profile 2: data(4,3,416,416)+img_info(4,4)

Profile 3: data(8,3,416,416)+img_info(8,4)

Dependencies and Restrictions

• Restrictions:
  • Networks that contain dynamic-shape operators (middle layers of the network with unfixed shape) are not supported.
  • If you have set dynamic batch sizes as well as dynamic AIPP (by setting --insert_op_conf):

    In your inference code, call the aclmdlSetInputAIPP API to set dynamic AIPP parameters. Ensure that batch_size is set to the allowed maximum batch size profile. For details about the API, see ""aclmdlSetInputAIPP"".

  • The offline model generated with this option is configured with the dynamic batch size feature, which may result in structural differences compared with models generated without this option, leading to different inference performance.
• API usage:

  If this option is used to set the dynamic batch size during model conversion, you need to perform the following operations before calling the model execution APIs to run an application project for inference:

  • Use the aclmdlSetDynamicBatchSize API to set the real batch size profile.
  • If aclmdlSetDynamicBatchSize is not used, the maximum value in the batch size range is assigned by default during model execution.

  For details about the API, see ""aclmdlSetDynamicBatchSize"".