--dynamic_batch_size

Applicable Products

All processors

Description

Sets dynamic batch size profiles, which applies to the scenario where the number of images or sentences processed per inference batch is unfixed.

In certain inference scenarios, for example, where an object recognition network is executed after object detection, the batch size of the network input is dynamic due to the unfixed number of objects. It would be a great waste of computing resources to perform each inference using the maximum batch size or image size. Thanks to ATC's support for dynamic batch sizes and dynamic image sizes, you can include this option and the --dynamic_image_size option in your model conversion command to set the supported batch size or image size profiles, respectively.

After model conversion, an input is added to the generated offline model (.om file). During model inference, the specific batch size (batch_size) is provided through this new input. Assume that the batch size (batch_size) of input a is dynamic. In the generated offline model, input b is added to describe the specific batch size of input a.

See Also

• Use this option together with --input_shape. This option is mutually exclusive with --dynamic_image_size, --input_shape_range, and --dynamic_dims. Only the scenario where N is in the first dimension of the shape is supported. That is, the first dimension of the shape is set to -1. If N is not in the first dimension, use --dynamic_dims to set it.

Arguments

Argument: Batch size profiles, for example, "1,2,4,8".

Format: Enclose the whole argument in double quotation marks (""), and separate the batch sizes with commas (,).

Restrictions:

• For the following products, at least two profiles must be set, and a maximum of 100 profiles are supported, that is, (1, 100]. The recommended value range for each profile is [1, 2048].

  Atlas A3 training products / Atlas A3 inference products

  Atlas A2 training products / Atlas A2 inference products

  Atlas 200I/500 A2 inference products

  Atlas inference products

  Atlas training products

• For the Ascend 950PR / Ascend 950DT , at least two profiles must be set, and a maximum of 256 profiles are supported, that is, (1, 256]. The recommended value range for each profile is [1, 2048].
• In the scenario where you have set too large batch sizes or too many batch size profiles, you are advised to run the swapoff -a command to disable the use of swap space as memory to prevent slow operating environment.

Suggestions and Benefits

• Too large batch sizes or too many batch size profiles will cause model conversion failures.
• For computer vision (CV) networks, the recommended values of --dynamic_batch_size are 8 and 16. In this scenario, the network achieves better performance compared with those using a single batch size. (Note that 8 and 16 are only recommended values; the final values should be determined based on actual test results.)
• For optical character recognition (OCR) or natural language processing (NLP) networks, the recommended value of --dynamic_batch_size is an integer multiple of 16.

Example

--input_shape="data:-1,3,416,416;img_info:-1,4"  --dynamic_batch_size="1,2,4,8"

-1 in --input_shape indicates that the dynamic batch size is set. The input supports the following shape profiles at ATC build time:

Profile 0: data(1,3,416,416)+img_info(1,4)

Profile 1: data(2,3,416,416)+img_info(2,4)

Profile 2: data(4,3,416,416)+img_info(4,4)

Profile 3: data(8,3,416,416)+img_info(8,4)

Dependencies and Restrictions

• Option Usage:
  • Networks that contain dynamic-shape operators (middle layers of the network with unfixed shape) are not supported.
  • If you have set dynamic batch sizes as well as dynamic AIPP (via --insert_op_conf):

    During inference, call the aclmdlSetInputAIPP API to set dynamic AIPP parameters. Ensure that batch_size is set to the maximum batch size profile. For details about the API, see aclmdlSetInputAIPP.

  • The offline model generated with this option enables the dynamic batch size feature, which may introduce structural differences compared with models from fixed batch size scenarios and result in differences in inference performance.
• API Usage:

  If this option is used to set the dynamic batch size during model conversion, you need to perform the following operations before calling the model execution APIs to run an application project for inference:

  • Use the aclmdlSetDynamicBatchSize API to set the real batch size profile.
  • If aclmdlSetDynamicBatchSize is not used, the maximum value in the batch size range is assigned by default during model execution.

  For details about the API, see aclmdlSetDynamicBatchSize.