Documentation
Download

Dynamic Batch/Dynamic Image Size/Dynamic Dimension (Setting Multi-Dimension Profiles)

API Call Sequence

The model inference process in the dynamic shape input scenario is similar to that in the Model Management scenario. Both of them involve initialization and deinitialization, runtime resource allocation and release, model building, model loading, model execution, and model unloading.

This section focuses on the differences between the two scenarios.

  1. During model building: You need to set the dynamic batch size, dynamic image size, and dynamic dimensions (ND format).

    If the dynamic batch feature is involved during model inference, call the PyACL API to set the batch size used for model inference. The batch size supported by the model has been configured during model building (by using the dynamic_batch_size parameter of ATC).

    If the dynamic image size feature is involved during model inference, call the PyACL API to set the image size. The image size supported by the model has been set during model building (by using the dynamic_image_size parameter of ATC).

    If the dynamic dimensions (ND format only) feature is involved during model inference, call the PyACL API to set the dimensions. The dimensions supported by the model have been configured during model building (by using the dynamic_dims parameter of ATC).

    After model building, the inputs of dynamic batch size, dynamic image size, and dynamic dimensions are added to the generated OM model. During model inference, the input values are provided.

    Assume that the batch size of input a is dynamic. In the generated OM model, input b is added to describe the batch size of input a.

    • When the model is executed, prepare the data structure of input a. For details, see Preparing Input/Output Data Structure for Model Execution.

      The shape of the input tensor supports multiple profiles, which can be determined only before model execution. Therefore, you are advised to call the acl.mdl.get_input_size_by_index and acl.mdl.get_output_size_by_index APIs to obtain the memory size required by the input and output. This API obtains the memory of the maximum profile to ensure that the memory is sufficient. After the model is executed, you can call the acl.mdl.get_output_dims API to obtain the actual dimension information of the model output tensor.

    • For details about how to prepare and set the data structure of input b, see 2.

    For details about the ATC parameters, see ATC.

  2. Before model inference
    • Prepare the data structures of the inputs of dynamic batch size, dynamic image size, and dynamic dimensions.
      1. Before allocating memory for the dynamic batch size, dynamic image size, or dynamic dimension input, call the acl.mdl.get_input_index_by_name API to obtain the index that identifies the input in the model based on the input name (fixed at ascend_mbatch_shape_data).
      2. Call acl.mdl.get_input_size_by_index to obtain the input memory size based on the index.
      3. Pass the size obtained in 2.b to the acl.rt.malloc call to allocate memory.

        Data in the memory does not need to be set after memory allocation (otherwise, services may be abnormal). After the APIs described in 2.b are called, the system automatically sets the data in the memory.

      4. Call the acl.create_data_buffer API to create data of the aclDataBuffer type, which is used to store the memory address and memory size of the dynamic batch size, dynamic image size, and dynamic dimension input data.
      5. Call the acl.mdl.create_dataset API to create data of the aclmdlDataset type, and call the acl.mdl.add_dataset_buffer API to add data of the aclDataBuffer type to the data of the aclmdlDataset type.
    • Set the dynamic batch size, dynamic image size, and dynamic dimensions.
      Figure 1 API call sequence
      1. Call the acl.mdl.get_input_index_by_name API to obtain the index that identifies the input in the model based on the input name (fixed at ascend_mbatch_shape_data).
      2. Set the dynamic batch size, dynamic image size, and dynamic dimensions.
        • Call the acl.mdl.set_dynamic_batch_size API to set the dynamic batch size.

          The configured batch size must be among the batch size profiles set during model building.

          You can also call the acl.mdl.get_dynamic_batch API to obtain the number of batch size profiles supported by a specified model and the batch size of each batch size profile.

        • Call acl.mdl.set_dynamic_hw_size to set the dynamic image size.

          The configured image size must be among the image size profiles set during model building.

          You can also call the acl.mdl.get_dynamic_hw API to obtain the number of resolution profiles supported by a specified model and the width and height of each profile.

        • Call acl.mdl.set_input_dynamic_dims to set the dynamic dimensions.

          The configured dimensions must be among the dimension profiles set during model building.

          You can also call the acl.mdl.get_input_dynamic_dims API to obtain the number of dynamic dimension size profiles supported by a specified model and the size of each profile.

Sample Code for Dynamic Batch

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to use.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
 
# 1. Load the model and set dynamic batch size profiles.
# ......

# 2. Create data of type aclmdlDataset to describe the inputs and outputs of the model.
# ......

# 3. Customize a function for setting dynamic batch size profiles.
def model_set_dynamic_info():
    # 3.1 Obtain the indexes of the input with dynamic batch size. The input name is fixed to ascend_mbatch_shape_data.
    index, ret = acl.mdl.get_input_index_by_name(model_desc, "ascend_mbatch_shape_data")
    # 3.2 Set the batch size. model_id indicates the ID of a successfully loaded model, input indicates data of type aclmdlDataset, and index indicates the index of the input with dynamic batch size.
    batch_size = 8
    ret = acl.mdl.set_dynamic_batch_size(model_id, input, index, batch_size)
    # ......

# 4. Customize a function and execute the model.
def model_execute(index):
    # 4.1 Call the customized function to set dynamic batch size profiles.
    ret = model_set_dynamic_info()
    # 4.2 Execute the model. model_id indicates the ID of a successfully loaded model. input and output indicate the inputs and outputs of the model.
    ret = acl.mdl.execute(model_id, input, output)
    # ......

# 5. Process the model inference result.
# ......

Sample Code for Dynamic Image Size

Following the API calls, add exception handling branches and specify log printing of error and information levels. The following is a code snippet of key steps only, which is not ready to use.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
 
# 1. Load the model and set dynamic image size profiles.
# ......

# 2. Create data of type aclmdlDataset to describe the inputs and outputs of the model.
# ......

# 3. Customize a function for setting dynamic image size profiles.
def model_set_dynamic_info():
    # 3.1 Obtain the indexes of the inputs with dynamic image size. The input name is fixed to ascend_mbatch_shape_data.
    index, ret = acl.mdl.get_input_index_by_name(model_desc, "ascend_mbatch_shape_data")
    # 3.2 Set the image size. model_id indicates the ID of a successfully loaded model, input indicates data of type aclmdlDataset, and index indicates the index of the input with dynamic image size.
    height = 224
    width = 224
    ret = acl.mdl.set_dynamic_hw_size(model_id, input, index, height, width)
    # ......

# 4. Customize a function and execute the model.
def model_execute(index):
    # 4.1 Call the customized function to set dynamic image size profiles.
    ret = model_set_dynamic_info()
    # 4.2 Execute the model. model_id indicates the ID of a successfully loaded model. input and output indicate the inputs and outputs of the model.
    ret = acl.mdl.execute(model_id, input, output)
    # ......

# 5. Process the model inference result.
# ......

Sample Code for Dynamic Dimensions (ND Format)

Add an exception handling branch following the API calls. The following is a code snippet of key steps only, which is not ready to use. 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
 
import acl
# ......

# 1. Load the model and set dynamic dimension profiles.
# ......

# 2. Prepare the model description information model_desc, model input data, and model output data.
# ......

# 3. Define a function for setting dynamic dimension profiles.
def model_set_dynamic_info():
    # 3.1 Obtain the indexes of the inputs with dynamic dimensions. The input name is fixed to ascend_mbatch_shape_data.
    index, ret = acl.mdl.get_input_index_by_name(model_desc, "ascend_mbatch_shape_data")
# 3.2 Set the profile information, including the number of dimensions (dimCount) and the value of each dimension. model_id indicates the ID of the model that is successfully loaded, input indicates the data of the aclmdlDataset type, and index indicates the input index of the dynamic dimension input.
    current_dims = {'name': '', 'dimCount': 4, 'dims': [8, 3, 224, 224]}
    ret = acl.mdl.set_input_dynamic_dims(model_id, input, index, current_dims)
    # ......

# 4. Customize a function and execute the model.
def model_execute(index):
    # 4.1 Call the customized function to set dynamic dimension profiles.
    ret = model_set_dynamic_info()
    # 4.2 Execute the model. model_id indicates the ID of a successfully loaded model. input and output indicate the inputs and outputs of the model.
    ret = acl.mdl.execute(model_id, input, output)
    # ......

# 5. Process the model inference result.
# ......
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details