Dynamic Shape

ac_parallel_enable

Sets whether to allow AI CPU operators and AI Core operators to run in parallel in a dynamic shape graph.

In a dynamic shape graph, when this parameter is enabled, the system automatically identifies AI CPU operators that can be concurrently executed with the AI Core operators in the graph. Operators of different engines are distributed to different flows to implement parallel execution among multiple engines, improving resource utilization and dynamic shape execution performance.

Parameter values:

1: AI CPU operators and AI Core operators are allowed to run in parallel.
0 (default): AI CPU operators are not separately distributed.

Configuration example:

{"ac_parallel_enable", "1"};

Mandatory/Optional: optional

Effective level: global

ge.dynamicDims

Sets the dynamic dimension profile in ND format. This parameter applies to the scenario where any dimension is processed each time during inference. This parameter must be used together with ge.inputShape. For details about the example, see Graph Build and Run.

Parameter value: Set it in the "dim1,dim2,dim3;dim4,dim5,dim6;dim7,dim8,dim9" format.

Format: Enclose all profiles in double quotation marks (""), separate the profiles by semicolons (;), and separate values within each profile by commas (,). The dimension size values match the -1 placeholders in the ge.inputShape parameter with ordering preserved, and the number of -1 placeholders equals the number of dimension sizes of each profile. More than one dynamic dimension profile must be provided.

Restrictions:

For the following products, the profile range is (1, 100]. At least two profiles must be set, and a maximum of 100 profiles are supported. Three to four profiles are recommended.
Atlas A3 training products / Atlas A3 inference products

Atlas A2 training products / Atlas A2 inference products

Atlas training products

Atlas inference products
For the Ascend 950PR / Ascend 950DT , the profile range is (1, 256]. At least two profiles must be set, and a maximum of 256 profiles are supported. Three to four profiles are recommended.

Examples:

If the model has only one input:

{"ge.inputShape", "data:1,-1,-1"},
{"ge.dynamicDims", "1,2;3,4;5,6;7,8"}
// During graph running, the supported shape of the data operator is 1,1,2; 1,3,4; 1,5,6; 1,7,8.

If the network model has multiple inputs:
The dimension size values match the -1 placeholders in the argument with ordering preserved, and the number of -1 placeholders equals the number of dimension sizes of each profile. Assume that a network model has three inputs: data (1,1,40,T), label (1,T), and mask (T,T), where T indicates a dynamic dimension. A configuration example is as follows:
```
{"ge.inputShape", "data:1,1,40,-1;label:1,-1;mask:-1,-1" },
{"ge.dynamicDims", "20,20,1,1;40,40,2,2;80,60,4,4"}
// During graph running, the following input dims combinations are supported:
// Profile 0: data(1,1,40,20)+label(1,20)+mask(1,1)
// Profile 1: data(1,1,40,40)+label(1,40)+mask(2,2)
// Profile 2: data(1,1,40,80)+label(1,60)+mask(4,4)
```

Mandatory/Optional: optional

Effective level: session and graph

ge.dynamicNodeType

Sets the type of a dynamic input node.

0: dataset input
1: placeholder input

Only one type of dynamic inputs is allowed, dataset or placeholder.

Configuration example:

{"ge.dynamicNodeType", "0"};

Mandatory/Optional: optional

Effective level: session and graph

ge.exec.hostSchedulingMaxThreshold

Sets the maximum threshold to enable dynamic shape scheduling when a static small graph (root graph) is executed. The default value is 0. It is recommended that this parameter be used in foundation model scenarios.

If the number of static root graph nodes is less than the maximum threshold, dynamic shape scheduling is used. For foundation models, this mode saves stream resources.
If the number of static root graph nodes is greater than the maximum threshold, the original process remains unchanged.

Note: If the static root graph node contains subgraphs, this parameter does not take effect.

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.static_model_ops_lower_limit

Sets the minimum number of operators for a static subgraph during dynamic shape splitting..

Parameter values: The value ranges from -1 to positive infinity. If other values are used, an error is reported. The default value is 4.

Value ≥ 0:
- If the number of operators is less than the lower limit, static subgraphs are not partitioned separately but are merged into dynamic graphs for dynamic execution.
- If the number of operators is greater than or equal to the lower limit, the original process remains unchanged (static sub-graphs are still partitioned).
-1: All operators are executed through dynamic graphs.

For example, if there are four operators in a static subgraph and this parameter is set to 10, static subgraphs are not partitioned separately but are executed through dynamic graphs.

Mandatory/Optional: optional

Effective level: graph

ge.inputShape

Sets the shape of the model input. For online inference or training, this parameter is mandatory only in dynamic profile scenarios and does not take effect in non-dynamic profile scenarios even if it is configured.

In dynamic profile scenarios, if one or more dimension values of the input data in the original model are not fixed, the model can be converted by setting the shape profile.

Parameter values:

The model has a non-static shape.
When setting ge.inputShape, set the corresponding dimension value to -1. This parameter must be used together with ge.dynamicDims and ge.dynamicNodeType.
The model has a scalar + dynamic-profile shape.
If the model input has both scalar shape and dynamic-profile shape, the scalar input must be configured. For example, if a model has three inputs: A:[-1,c1,h1,w1], B:[], and C:[n2,c2,h2,w2], the shape information is "A:-1,c1,h1,w1;B:;C:n2,c2,h2,w2". Scalar input B must be configured.

Configuration example:

Non-static shape
For details about how to set profiles for a specified dimension, see ge.dynamicDims.
Scalar + dynamic-profile shape
Shape is a scalar input, which is mandatory. For example, if the model has three inputs and the shape information is A:[-1,32,208,208], B:[], and C:[16,64,208,208], the configuration example is as follows (A is the dynamic profile input, and the dynamic dimensions profile is used):
```
{"ge.inputShape", "A:-1,32,208,208;B:;C:16,64,208,208"}, 
{"ge.dynamicDims", "1;2;4"} 
```

Mandatory/Optional: optional

Effective level: session and graph

Parent topic: Options