Concepts and Restrictions

**Table 1** Concepts
Term	Description
Dynamic batch/image size	In some scenarios, the batch size or resolution input to the model each time is not fixed. For example, if the target recognition network is executed after the target is detected, the batch size input to the target recognition network is not fixed because the number of targets is not fixed. Dynamic batch: The batch size is not determinable until inference time. Dynamic image size: The image size (H x W) of each image is dynamically variable during inference.
Dynamic dimensions (ND format only)	To support the scenarios where the input dimension sizes are uncertain, such as the Transformer network, the dynamic dimensions feature for the ND format is supported. ND indicates any format with up to four dimensions.

For a single model, AIPP (static or dynamic) and dynamic dimensions (ND format) are mutually exclusive.
For the same model, you can select only one of the following methods:
- Call acl.mdl.set_dataset_tensor_desc to set the shape.
- Call the acl.mdl.set_dynamic_batch_size API to set the dynamic batch size.
- Call acl.mdl.set_dynamic_hw_size to set the dynamic image size.
- Call acl.mdl.set_input_dynamic_dims to set the dynamic dimensions.
You can allocate the output buffer for model inference based on the actual size of each profile. Alternatively, you can call the acl.mdl.get_output_size_by_index API to obtain the buffer size and then allocate the buffer. (You are advised to use this method to ensure that the memory is sufficient.)

If the dynamic AIPP and dynamic batch size are both enabled:
- The batchSize parameter passed to the acl.mdl.create_aipp call must be set to the allowed maximum profile.
- For data nodes that require dynamic AIPP, the corresponding input buffer must be allocated based on the maximum batch size profile.

If the dynamic AIPP and dynamic image size are both enabled:
- If the image cropping, resizing, or padding function is enabled in setting the dynamic AIPP parameters, the dynamic image size becomes unavailable.
- If the image cropping, resizing, or padding function is not enabled when the dynamic AIPP parameters are set, the dynamic AIPP parameters are used together with the dynamic image size. In this case, ensure that the width and height configured by calling acl.mdl.set_aipp_src_image_size are the same as those configured by calling acl.mdl.set_dynamic_hw_size.
- For data nodes that require dynamic AIPP processing, allocate the input buffer must be based on the maximum allowed image size.

When dynamic AIPP and dynamic shape input (setting the shape range) are used at the same time, the width and height of the output image of dynamic AIPP must be within the configured shape range.

If static AIPP and dynamic image size are both enabled, because the width and height of the input image are uncertain in the dynamic image size scenario, when the insert_op_conf parameter of the ATC tool is used to transfer the AIPP configuration file, the crop and padding functions cannot be enabled in the AIPP configuration file. Additionally, the src_image_size_w and src_image_size_h values in the configuration file need to be set to 0.

Parent topic: Model Inference with Dynamic-Shape Inputs