Operator and Graph Build

ge.aicoreNum

Sets the number of AI Cores used for operator build.

Parameter values: "integer 1|integer 2", separated by a vertical bar (|).

Scenario 1: For the following products, integer 1 indicates the number of Cube Cores in the AI Core used for operator build, and integer 2 indicates the number of Vector Cores in the AI Core used for operator build. Both integer 1 and integer 2 must be greater than 0 and less than or equal to the maximum numbers of Cube Cores and Vector Cores included in the AI processor.
Ascend 950PR / Ascend 950DT

Atlas A3 training products / Atlas A3 inference products

Atlas A2 training products / Atlas A2 inference products
Scenario 2: For the following products, only integer 1 needs to be configured in the format of "integer 1|", indicating the number of AI Cores used for operator build. If integer 2 is configured, it does not take effect.
Atlas inference products

Atlas training products

Restrictions:

For scenario 1:
You can view the maximum number of Cube Cores and Vector Cores contained in different AI processors in the "${INSTALL_DIR}/<arch>-linux/data/platform_config/xxx.ini" file. As shown in the following figure, there are 24 Cube Cores and 48 Vector Cores on the AI processor.
```
[SoCInfo]
# Use the default parameter values, which are the maximum values.
ai_core_cnt=24
cube_core_cnt=24
vector_core_cnt=48
```
For scenario 2:
You can view the maximum number of AI Cores contained in different AI processors in the "${INSTALL_DIR}/<arch>-linux/data/platform_config/xxx.ini" file. As shown in the following figure, there are 10 AI Cores on the AI processor.
```
[SoCInfo]
# Use the default parameter value, which indicates the maximum number of AI Cores.
ai_core_cnt=10
vector_core_cnt=8
```
If the operator build cache function is enabled (ge.op_compiler_cache_mode set to enable or force; default value: enable) and this parameter is configured, this parameter takes effect only during the first compilation. To make this parameter take effect during non-initial build, you need to clear the cache of the build disk.

Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann. <arch> indicates the OS architecture, and xxx varies depending on the product.

Configuration example:

Configuration example for scenario 1:
```
{"ge.aicoreNum", "24|48"};
```

Configuration example for scenario 2:

{"ge.aicoreNum", "10|"};
or
{"ge.aicoreNum", "10"};

Mandatory/Optional: optional

Effective level: global and session

Relationships between AI Cores, Cube Cores, and Vector Cores:

The definition of a Core helps you better understand the relationships between AI Cores, Cube Cores, and Vector Cores. A Core is a compute core with an independent scalar compute unit. Generally, the scalar compute unit provides multiple functions for the compute core, such as the single instruction multiple data (SIMD) instruction dispatch. The scalar compute unit is also called the intra-core scheduling unit. The AI data processing core unit varies with products. Currently, there are the following types:

The AI data processing core unit is an AI Core:
- In an AI Core, a Cube and a Vector share a Scalar scheduling unit, for example, Atlas training products .
- In an AI Core, a Cube and a Vector have their own Scalar scheduling units, which are also called a Cube Core and a Vector Core. In this case, a Cube Core and a group of Vector Cores are defined as an AI Core. The number of AI Cores is usually calculated based on the number of Cube Cores, for example, Atlas A2 training products / Atlas A2 inference products .
The AI data processing core units are AI Cores and independent Vector Cores. The AI Cores and Vector Cores have independent Scalar scheduling units, for example, Atlas inference products .

ge.AllowMultiGraphParallelCompile

Controls the capability of allowing multiple threads to compile multiple graphs in parallel in the same session. If this parameter is set to 1, the variable format cannot be converted. For details, see the description of ge.exec.variable_acc.

Parameter values:

0 (default): One thread can be used to compile multiple graphs in the same session.
1: Multiple threads can be used to compile multiple graphs concurrently in the same session.

Restrictions:

If this parameter is set to 1, ge.exec.variable_acc cannot be set to True. Otherwise, an error is reported during verification.
If this parameter is set to 1, an error is reported immediately when resource operators cause re-compilation of other graphs.

Configuration example:

{"ge.AllowMultiGraphParallelCompile", "1"};

Mandatory/Optional: optional

Effective level: global and session

ge.graph_compiler_cache_dir

Sets the disk cache directory for graph build. This parameter is used together with ge.graph_key. This function takes effect only when both ge.graph_compiler_cache_dir and ge.graph_key are not empty.

The configured cache directory must exist. Otherwise, the compilation fails.

After a graph is changed, the original cache file is unavailable. You need to manually delete the cache file from the cache directory or modify ge.graph_key to re-compile and generate a cache file.

For details about other restrictions and usage methods, see Graph Build Cache.

Mandatory/Optional: optional

Effective level: session

ge.graph_key

Specifies the unique ID of a graph. The value contains a maximum of 128 characters, including only letters, digits, underscores (_), and hyphens (-).

Mandatory/Optional: optional

Effective level: graph

ge.op_compiler_cache_mode

Sets the disk cache mode for operator build.

Parameter values:

enable (default): Cache is enabled for operator build. After this function is enabled, operators with the same compilation parameters and operator parameters will not be built repeatedly, improving the compilation speed.
force: Cache is enabled for operator build, with the cache forcibly refreshed. The existing cache is cleared up before the operator is re-built and added to the cache. The implementation with the force mode used is different from that when the enable mode is used. For example, for Python changes, dependency library changes, or repository changes after operator optimization, you need to set this parameter to force to clear up the existing cache and then change it to the enable mode to prevent the cache from being forcibly refreshed during each compilation.
disable: Cache is disabled for operator build, and the operator is re-built.

Restrictions:

To specify the disk cache path for operator build, use this parameter together with ge.op_compiler_cache_dir.
If it is set to force, the existing cache will be cleared. Therefore, it is not recommended for parallel program compilation, as this may cause the cache used by other models to be cleared, resulting in failures.
disable or force is recommended for publishing the final model.
If the repository changes after operator tuning, set this parameter to force to refresh the cache and then set it to enable for re-compilation. Otherwise, the new tuning repository cannot be applied, and the tuning application fails to be executed.
When the debugging function is enabled:
- If ge.opDebugLevel is set to a non-zero value, the ge.op_compiler_cache_mode parameter configuration does not take effect, the operator build cache function is disabled, and all operators are re-built.
- If op_debug_config is not empty and op_debug_list is not configured, the ge.op_compiler_cache_mode parameter configuration does not take effect, the operator build cache function is disabled, and all operators are re-built.
- If op_debug_config is not empty and op_debug_list is configured in the configuration file:
  - For operators in the list, ignore the ge.op_compiler_cache_mode parameter configuration and continue with the re-compilation.
  - For operators out of the list, if ge.op_compiler_cache_mode is set to enable or force, the cache function is enabled. If it is set to disable, the cache function is disabled and the operators are re-built.
When you enable the operator build cache function, set the disk space of the cache folder by using the configuration file (with the op_cache.ini file automatically generated in the path specified by ge.op_compiler_cache_dir after operator build) or using environment variables.
1. Using the op_cache.ini configuration file:
  If the op_cache.ini file does not exist, manually create it. Open the file and add the following information:
```
# Configure the file format (required). The automatically generated file contains the following information by default. When manually creating a file, enter the following information:
[op_compiler_cache]
# Limit the disk space of the cache folder on a chip, in MB. The value must be an integer and defaults to 500.
max_op_cache_size=500
# Set the percentage of the cache space to be reserved. The value range is [1, 100]. The default value is 50. For example, 80 indicates that when the cache space is insufficient, cache files will be deleted to keep 80% of the cache space available.
remain_cache_size_ratio=50    
```
  - The op_cache.ini file takes effect only when the values of max_op_cache_size and remain_cache_size_ratio in the preceding file are valid.
  - If the size of the build cache file exceeds the value of max_op_cache_size and the cache file is not accessed for more than half an hour, the cache file will be aged. (Operator build will not be interrupted due to the size of the build cache file exceeding the set limit. Therefore, if max_op_cache_size is set to a small value, the size of the actual build cache file may exceed the configured value.)
  - To disable the build cache aging function, set max_op_cache_size to -1. In this case, the access time is not updated when the operator cache is accessed, the operator build cache is not aged, and the default disk space of 500 MB is used.
  - If multiple users use the same cache path, you are advised to use the configuration file to set the cache path. In this scenario, the op_cache.ini file affects all users.
2. Using environment variables
  In this scenario, the environment variable ASCEND_MAX_OP_CACHE_SIZE is used to limit the storage space of the cache folder of a chip. When the build cache space reaches the specified value and the cache file is not accessed for more than half an hour, the cache file is aged. The environment variable ASCEND_REMAIN_CACHE_SIZE_RATIO is used to set the ratio of the cache space to be reserved.
  
  A configuration example is provided as follows:
```
# The ASCEND_MAX_OP_CACHE_SIZE environment variable defaults to 500, in MB. The value must be an integer.
export ASCEND_MAX_OP_CACHE_SIZE=500
# The value range of the ASCEND_REMAIN_CACHE_SIZE_RATIO environment variable is [1, 100]. The default value is 50, in percentage. For example, 80 indicates that when the cache space is insufficient, cache files will be deleted to keep 80% of the cache space available.
export ASCEND_REMAIN_CACHE_SIZE_RATIO=50
```
  - The argument configured through environment variables takes effect only for the current user.
  - To disable the build cache aging function, set the environment variable ASCEND_MAX_OP_CACHE_SIZE to -1. In this case, the access time is not updated when the operator cache is accessed, the operator build cache is not aged, and the default disk space of 500 MB is used.
If both the op_cache.ini file and the environment variable are configured, the configuration items in the op_cache.ini file are read first. If both the op_cache.ini file and the environment variable are not configured, the system's default values (500 MB disk space and 50% reserved cache space) are read.

Configuration example:

{"ge.op_compiler_cache_mode", "enable"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.op_compiler_cache_dir

Sets the disk cache directory for operator build.

Format: The path can contain letters, digits, underscores (_), hyphens (-), and periods (.).

Default value: $HOME/atc_data

If the specified directory exists and is valid, a kernel_cache subdirectory is automatically created. If the specified directory does not exist but is valid, the system automatically creates this directory and the kernel_cache subdirectory.
Do not store other self-owned content in the default cache directory. The self-owned content will be deleted together with the default cache directory during software package installation or upgrade.
The non-default cache directory specified by this parameter cannot be deleted. The directory will not be deleted during software package installation or upgrade.
In addition to ge.op_compiler_cache_dir, the ASCEND_CACHE_PATH environment variable can be used to set the disk cache directory for operator build. The priorities of the configuration methods are as follows: ge.op_compiler_cache_dir > ASCEND_CACHE_PATH > default storage directory.

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.optimizationSwitch

Sets whether to enable the fusion pattern (pass) during operator build.

The difference between this parameter and ge.fusionSwitchFile is as follows: This parameter applies to all patterns. It can be used to specify a fusion pattern without a JSON file. ge.fusionSwitchFile can only be used to disable the graph fusion and UB fusion patterns, and a JSON file needs to be configured separately. If both parameters are set and the same fusion pattern is configured, the setting of ge.optimizationSwitch takes precedence.

Parameter value: Passname1:on;Passname2:off. Multiple key-value pairs can be concatenated. The key is the pass name, and value can be set to on (enabled) or off (disabled). Case-sensitive matching is not supported. Multiple groups of configurations are separated by semicolons (;). For details about the fusion patterns that can be configured, see Fusion Pattern List.

Configuration example:

{"ge.optimizationSwitch", "Passname1:on;Passname2:off"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.topoSortingMode

Specifies the traversal mode when you compile operators in graph mode. It is mainly used in online inference scenarios.

Parameter values:

0: Breadth First Search (BFS)
1 (default): Depth First Search (DFS)
2: Reverse DFS (RDFS)
3: Stable RDFS. For existing operators in the graph, the computation sequence is not changed. For new operators in the graph, RDFS is used.

Configuration example:

{"ge.topoSortingMode", "1"};

Mandatory/Optional: optional

Effective level: global, session, and graph

Parent topic: Options