Precision Comparison

ge.bufferOptimize

Sets whether to enable buffer optimization.

Parameter values:

l1_optimize: Enables L1 optimization. This option is invalid in the current version and is equivalent to off_optimize.
l2_optimize (default): Enables L2 optimization.
off_optimize: Disables buffer optimization.

Suggestions:

You are advised to enable buffer optimization. This function can improve computing efficiency and performance. However, it is possible that your model contains an operator that is not yet covered by the current implementation, which affects the precision. You can disable data buffer optimization when the precision is affected. If the precision meets the requirement after the data cache optimization function is disabled, identify the faulty operator and contact technical support for further analysis and resolution. After the operator problem is resolved, you are advised to enable the data cache optimization function.

Configuration example:

{"ge.bufferOptimize", "l2_optimize"};

Mandatory/Optional: optional

Effective level: session and graph

ge.exec.enableDump

Sets whether to enable the dump function.

1: The dump function is enabled. The dump file path is read from dump_path. If dump_path is set to None, an exception occurs.
0 (default): The dump function is disabled.

Configuration example:

{"ge.exec.enableDump", "0"};

Mandatory/Optional: optional

Effective level: global and session

Restrictions:

This parameter cannot be used together with ge.exec.enableDumpDebug in the global scenario or in the same session.
If either ge.exec.enableDump or ge.exec.enableDumpDebug is set to 1 and ge.exec.enable_exception_dump is set to 1 (indicating that common ExceptionDump function is enabled):
- For dynamic-shape networks, only ge.exec.enable_exception_dump takes effect.
- For static-shape networks, ge.exec.enable_exception_dump and either of ge.exec.enableDump and ge.exec.enableDumpDebug take effect.

ge.exec.enableDumpDebug

Sets whether to enable overflow detection.

1: Overflow detection is enabled. The dump file path is read from ge.exec.dumpPath. If ge.exec.dumpPath is set to None, an exception occurs.
0 (default): Overflow detection is disabled.

Configuration example:

{"ge.exec.enableDumpDebug", "0"};

Mandatory/Optional: optional

Effective level: global and session

Restrictions:

This parameter cannot be used together with ge.exec.enableDump in the global scenario or in the same session.
If either ge.exec.enableDump or ge.exec.enableDumpDebug is set to 1 and ge.exec.enable_exception_dump is set to 1 (indicating that common ExceptionDump function is enabled):
- For dynamic-shape networks, only ge.exec.enable_exception_dump takes effect.
- For static-shape networks, ge.exec.enable_exception_dump and either of ge.exec.enableDump and ge.exec.enableDumpDebug take effect.

ge.exec.dumpData

Sets the type of operator content to be dumped.

tensor (default): Operator data is dumped.
stats: Operator statistics are dumped and the result is saved in CSV format. As the operator data amount is large in most cases, you can try to dump the operator statistics.

Configuration example:

{"ge.exec.dumpData", "tensor"};

Mandatory/Optional: optional

Effective level: global and session

ge.exec.dumpDebugMode

Sets the overflow detection mode.

Parameter values:

aicore_overflow: detects AI Core operator overflow, that is, detecting whether abnormal extreme values (such as 65500, 38400, and 51200 in float16) are output with normal inputs. Once such fault is detected, analyze the cause of the overflow and modify the operator implementation based on the network requirements and operator logic.
atomic_overflow: detects Atomic Add overflow, for checking modules involved in floating-point computing (such as SDMA) in addition to AI Core.
all: detects overflow of both AI Core operators and Atomic Add.

For the following products, this parameter can only be set to all:

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Configuration example:

{"ge.exec.dumpDebugMode", "all"};

Mandatory/Optional: optional

Effective level: global and session

ge.exec.dumpLayer

Specifies the operators that need to be dumped. The value must be operator names, which are separated by spaces.

If the input of the specified operator involves the data operator, the data operator information is also dumped.

Configuration example:

{"ge.exec.dumpLayer", "layer1 layer2 layer3"};

Mandatory/Optional: optional

Effective level: global and session

ge.exec.dumpMode

Sets the dump mode to specify whether the operator input or output is dumped. The values are as follows:

input: dumps only operator inputs.
output (default): dumps only operator outputs.
all: dumps both operator inputs and outputs.

Configuration example:

{"ge.exec.dumpMode", "input"};

Restrictions:

If this parameter is set to all, the input data of some operators, such as collective communication operators HcomAllGather and HcomAllReduce, will be modified during execution. The system dumps the operator input before operator execution and dumps the operator output after operator execution. In this way, the dumped input and output data of the same operator is flushed to disks separately, and multiple dump files are generated. After parsing the dump files, you can determine whether the data is an input or output based on the file content.

Mandatory/Optional: optional

Effective level: global and session

ge.exec.dumpPath

Sets the path for storing the dump file. This parameter is required when dump and overflow detection are enabled.

Create the specified path in advance in the environment (either container or host) where training is performed. The running user configured during installation must have the read and write permissions on this path. The path can be an absolute path or a relative path (relative to the current path when the command is executed).

An absolute path starts with a slash (/), for example, /home/HwHiAiUser/output.
A relative path starts with a directory name, for example, output.

The dump data file is generated in the specified directory, that is, the {dump_path}/{time}/{deviceid}/{model_name}/{model_id}/{data_index} directory. For example, if dump_path is set to /home/HwHiAiUser/output, the dump data file is stored in the /home/HwHiAiUser/output/20200808163566/0/ge_default_20200808163719_121/11/0 directory.

Mandatory/Optional: optional

Effective level: global and session

ge.exec.dumpStep

Specifies the iterations whose dump data is to be collected. It defaults to None, indicating that all iterations are dumped.

Separate multiple iterations using vertical bars (|), for example, 0|5|10. You can also use hyphens (-) to specify the iteration range, for example, 0|3-5|10.

Configuration example:

{"ge.exec.dumpStep", "None"};

Mandatory/Optional: optional

Effective level: global and session

ge.fusionSwitchFile

Sets the directory (including the file name) of the configuration file for the fusion pattern (including graph fusion and UB fusion) switch. The directory can contain letters, digits, underscores (_), hyphens (-), and periods (.).

The built-in graph fusion and UB fusion patterns are enabled by default. You can disable specified fusion patterns in the configuration file. Some fusion patterns cannot be disabled due to functionality restrictions. For the full list of fusion patterns that can be disabled, see Graph Fusion and UB Fusion Patterns.

The following is a template of the fusion_switch.cfg configuration file. The on indicates that a fusion pattern is enabled, and off indicates that a fusion pattern is disabled.

Example configuration file:

{
    "Switch":{
        "GraphFusion":{
            "RequantFusionPass":"on",
            "ConvToFullyConnectionFusionPass":"off",
            "SoftmaxFusionPass":"on",
            "NotRequantFusionPass":"on",
            "SplitConvConcatFusionPass":"on",
            "ConvConcatFusionPass":"on",
            "MatMulBiasAddFusionPass":"on",
            "PoolingFusionPass":"on",
            "ZConcatv2dFusionPass":"on",
            "ZConcatExt2FusionPass":"on",
            "TfMergeSubFusionPass":"on"
        },
        "UBFusion":{
            "TbePool2dQuantFusionPass":"on"
        }
    }
}

The Ascend 950PR/Ascend 950DT does not support UB fusion. There is no need to configure UBFusion.

In addition, users can disable fusion patterns by one click:

Example configuration file:

{
    "Switch":{
        "GraphFusion":{
            "ALL":"off"
        },
        "UBFusion":{
            "ALL":"off"
         }
    }
}

The Ascend 950PR/Ascend 950DT does not support UB fusion. There is no need to configure UBFusion.

Note:

Disabling certain fusion patterns may cause functional issues. Therefore, this one-click disable operation only disables some system-level fusion patterns instead of all fusion patterns.

To disable all fusion patterns except selected ones, refer to the following example.

Configuration file example:

{
    "Switch":{
        "GraphFusion":{
            "ALL":"off",
            "SoftmaxFusionPass":"on"
        },
        "UBFusion":{
            "ALL":"off",
            "TbePool2dQuantFusionPass":"on"
        }
    }
}

If UB fusion is not supported, UBFusion can be deleted from the configuration file.

Configuration example:

{"ge.fusionSwitchFile", "/home/test/fusion_switch.cfg"};

Mandatory/Optional: optional

Effective level: global, session, and graph

Parent topic: Options