Dynamic batch/dynamic image size/dynamic dimension (setting multiple dimension profiles)

API Call Sequence

The model inference process in the dynamic shape input scenario is similar to that in the Model Inference with Static-Shape Inputs scenario. Both of them involve initialization and deinitialization, runtime resource allocation and release, model building, model loading, model execution, and model unloading.

This section focuses on the differences between model inference and Model Inference with Static-Shape Inputs in the dynamic shape input scenario.

1. During model building: You need to set the dynamic batch size, dynamic image size, and dynamic dimensions (ND format).

   If the dynamic batch size feature is involved during model inference, call the acl API to set the batch size. The batch size supported by the model has been configured during model building. For example, the dynamic_batch_size parameter of ATC is used during model building. For details about the parameters, see Command-Line Options > Basic Options > Input Options > --dynamic_batch_size in ATC.

   If the dynamic image size feature is involved during model inference, call the acl API to set the image size. The image size supported by the model has been configured during model building. For example, the dynamic_image_size parameter of ATC is used during model building. For details about the parameters, see Command-Line Options > Basic Options > Input Options > --dynamic_image_size in ATC.

   If the dynamic dimensions (ND format only) are involved during model inference, call the acl API to set the dimensions. The dimensions supported by the model have been configured during model building, for example, the dynamic_dims parameter of ATC is used during model building. For details about the parameters, see --dynamic_dims in ATC.

   After model building, the inputs of dynamic batch size, dynamic image size, and dynamic dimensions are added to the generated OM model. During model inference, the input values are provided.

   Assume that the batch size of input a is dynamic. In the generated OM model, input b is added to describe the batch size of input a.

   • For details about the input data structure prepared during model execution, see Preparing Input/Output Data Structure for Model Execution.

     The shape of the input tensor supports multiple profiles, which can be determined only before model execution. Therefore, you are advised to call the aclmdlGetInputSizeByIndex and aclmdlGetOutputSizeByIndex APIs to obtain the memory size required by the input and output. This API obtains the memory size of the maximum profile, ensure that the memory is sufficient. After the model is executed, call aclmdlGetCurOutputDims to obtain the actual dimension information of the model output tensor.

   • For details about how to prepare the data structure and set the data of input b, see 2.
2. Before model inference
   • Prepare the data structures of the inputs of dynamic batch size, dynamic image size, and dynamic dimensions.
     1. Before allocating memory for the dynamic batch size, dynamic image size, or dynamic dimension input, call aclmdlGetInputIndexByName to obtain the index that identifies the input in the model based on the input name (fixed at ACL_DYNAMIC_TENSOR_NAME).
        

        ACL_DYNAMIC_TENSOR_NAME is a macro defined as follows:

        #define ACL_DYNAMIC_TENSOR_NAME "ascend_mbatch_shape_data"

     2. Call aclmdlGetInputSizeByIndex to obtain the input memory size based on the index.
     3. Call aclrtMalloc to apply for memory based on the size specified by 2.b.

        Do not initialize the buffer manually; otherwise, the service will be unavailable. After the API calls described in 2.b, the system automatically initializes the buffer.

     4. Create data of the aclDataBuffer type by calling aclCreateDataBuffer. The data is used to store the memory address and memory size of the dynamic batch size, dynamic image size, and dynamic dimension input data.
     5. Call aclmdlCreateDataset to create data of the aclmdlDataset type, and call aclmdlAddDatasetBuffer to add data of the aclDataBuffer type to the data of the aclmdlDataset type.
   • Set the dynamic batch size, dynamic image size, and dynamic dimensions.
     Figure 1 API call sequence
     
     1. Call aclmdlGetInputIndexByName to obtain the index that identifies the input in the model based on the input name (fixed at ACL_DYNAMIC_TENSOR_NAME).
     2. Set the dynamic batch size, dynamic image size, and dynamic dimensions.
        • Call aclmdlSetDynamicBatchSize to set the dynamic batch size.

          The configured batch size must be among the batch size profiles set during model building.

          You can also call aclmdlGetDynamicBatch to obtain the number of batch size profiles supported by a specified model and the batch size in each batch size profile.

        • Call aclmdlSetDynamicHWSize to set the dynamic image size.

          The configured image size must be among the image size profiles set during model building.

          You can also call aclmdlGetDynamicHW to obtain the number of resolution profiles as well as the width and height of each profile supported by a specified model.

        • Call aclmdlSetInputDynamicDims to set the dynamic dimension size.

          The configured dimensions must be among the dimension profiles set during model building.

          You can also call aclmdlGetInputDynamicDims to obtain the number of dynamic dimension size profiles and the size of each profile supported by a specified model.

Sample Code for Dynamic Batch

The following is a code example of key steps of model inference. It is for reference only and cannot be directly copied for compilation and running. After APIs are called, you need to add exception handling branches and record error logs and info logs.

You can click YOLOV3_dynamic_batch_detection_picture to obtain the sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

......

// 1. Load the model and set dynamic batch size after the model is successfully loaded.
......

// 2. Prepare the model description information modelDesc_, model input data input_, and model output data output_.
// ......

// 3. Customize a function to set dynamic batch size.
int  ModelSetDynamicInfo()
{
        size_t index;
       //3.1 Obtain the indexes the inputs with dynamic batch size. The input name is fixed to ACL_DYNAMIC_TENSOR_NAME.
        aclError ret = aclmdlGetInputIndexByName(modelDesc_, ACL_DYNAMIC_TENSOR_NAME, &index);
        //3.2 Set the batch size.
        //modelId_ indicates the ID of the model that is successfully loaded, input_ indicates data of the aclmdlDataset type, index indicates the input index of the dynamic batch input, and batchSize indicates the batch size (8 for example).
        uint64_t batchSize = 8;
        ret = aclmdlSetDynamicBatchSize(modelId_, input_, index, batchSize);
        // ......
}

// 4. Customize a function to execute the model.
int ModelExecute(int index)
{
        aclError ret;
        //4.1 Call the user-defined function to set the runtime batch size.
	ret = ModelSetDynamicInfo();
        //4.2 Execute the model. modelId_ indicates the ID of a successfully loaded model, input_ indicates the model inputs, and output_ indicates the model outputs.
        ret = aclmdlExecute(modelId_, input_, output_);
        // ......
}
// 5. Process the model inference result.
......

Sample Code for Dynamic Image Size

The following is a code example of key steps of model inference. It is for reference only and cannot be directly copied for compilation and running. After APIs are called, you need to add exception handling branches and record error logs and info logs.

You can click YOLOV3_dynamic_batch_detection_picture to obtain the sample.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

......

// 1. Load the model. After the loading is successful, set the dynamic image size.
......

// 2. Prepare the model description information modelDesc_, model input data input_, and model output data output_.
......

// 3. Customize a function to set the dynamic image size.
int  ModelSetDynamicInfo()
{
		size_t index;
                //3.1 Obtain the indexes of the inputs with dynamic image size. The input name is fixed to ACL_DYNAMIC_TENSOR_NAME.
		aclError ret = aclmdlGetInputIndexByName(modelDesc_, ACL_DYNAMIC_TENSOR_NAME, &index);
                //3.2 Set the image size. modelId_ indicates the ID of a successfully loaded model, input_ indicates data of type aclmdlDataset, and index indicates the index of the input with dynamic image size.
                uint64_t height = 224;
		uint64_t width = 224;
		ret = aclmdlSetDynamicHWSize(modelId_, input_, index, height, width);
                // ......
}

// 4. Customize a function to execute the model.
int ModelExecute(int index)
{
        aclError ret;
        //4.1 Call the user-defined function to set the runtime image size.
	ret = ModelSetDynamicInfo();
        //4.2 Execute the model. modelId_ indicates the ID of a successfully loaded model, input_ indicates the model inputs, and output_ indicates the model outputs.
        ret = aclmdlExecute(modelId_, input_, output_);
        // ......
}

// 5. Process the model inference result.
......

Sample Code for Dynamic Dimensions (ND Format Only)

The following is a code example of key steps of model inference. It is for reference only and cannot be directly copied for compilation and running. After APIs are called, you need to add exception handling branches and record error logs and info logs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

......

// 1. Load the model and set the dynamic dimension size after the model is successfully loaded.
......

// 2. Prepare the model description information modelDesc_, model input data input_, and model output data output_.
......

// 3. Customize a function to set the dynamic dimension size.
int  ModelSetDynamicInfo()
{
        size_t index;
        //3.1 Obtain the indexes of the inputs with dynamic dimensions. The input name is fixed to ACL_DYNAMIC_TENSOR_NAME.
        aclError ret = aclmdlGetInputIndexByName(modelDesc_, ACL_DYNAMIC_TENSOR_NAME, &index);
        //3.2 Set the runtime dimensions, including the dimension count (dimCount) and the size of each dimension. modelId_ indicates the ID of a successfully loaded model, input_ indicates data of type aclmdlDataset, and index indicates the index of the input with dynamic dimensions.
        aclmdlIODims currentDims;
        currentDims.dimCount = 4;
        currentDims.dims[0] = 8;
        currentDims.dims[1] = 3;
        currentDims.dims[2] = 224;
        currentDims.dims[3] = 224;
        ret = aclmdlSetInputDynamicDims(modelId_, input_, index, &currentDims);
        // ......
}

// 4. Customize a function to execute the model.
int ModelExecute(int index)
{
        aclError ret;
        //4.1 Call the user-defined function to set the runtime dimensions.
	ret = ModelSetDynamicInfo();
        //4.2 Execute the model. modelId_ indicates the ID of a successfully loaded model, input_ indicates the model inputs, and output_ indicates the model outputs.
        ret = aclmdlExecute(modelId_, input_, output_);
        // ......
}
// 5. Process the model inference result.
......