Session Configuration Options

graph_run_mode

Graph run mode.

• 0: online inference.
• 1 (default): training

Configuration example:

custom_op.parameter_map["graph_run_mode"].i = 1

Training/Online inference

session_device_id

Logical ID of a device. Setting this parameter allows you to run different models on multiple devices by executing a single training script.

You can create different sessions for different graphs and pass different session_device_id values.

Example:

config_0 = tf.ConfigProto()
custom_op = config_0.graph_options.rewrite_options.custom_optimizers.add()
custom_op.name = "NpuOptimizer"
custom_op.parameter_map["session_device_id"].i = 0
config_0.graph_options.rewrite_options.remapping = RewriterConfig.OFF
config_0.graph_options.rewrite_options.memory_optimization = RewriterConfig.OFF
with tf.Session(config=config_0) as sess_0:
    sess_0.run(...)

config_1 = tf.ConfigProto()
custom_op = config_1.graph_options.rewrite_options.custom_optimizers.add()
custom_op.name = "NpuOptimizer"
custom_op.parameter_map["session_device_id"].i = 1
config_1.graph_options.rewrite_options.remapping = RewriterConfig.OFF
config_1.graph_options.rewrite_options.memory_optimization = RewriterConfig.OFF
with tf.Session(config=config_1) as sess_1:
    sess_1.run(...)

config_7 = tf.ConfigProto()
custom_op = config_7.graph_options.rewrite_options.custom_optimizers.add()
custom_op.name = "NpuOptimizer"
custom_op.parameter_map["session_device_id"].i = 7
config_7.graph_options.rewrite_options.remapping = RewriterConfig.OFF
config_7.graph_options.rewrite_options.memory_optimization = RewriterConfig.OFF
with tf.Session(config=config_7) as sess_7:
    sess_7.run(...)

Training/Online inference

deterministic

Whether to enable deterministic computing. If enabled, the same output is generated if an operator is executed for multiple times with the same hardware and input.

The values are as follows:

• 0 (default): disables deterministic computing.
• 1: enables deterministic computing.

By default, deterministic computing does not need to be enabled, because it slows down operator execution and affects performance. If it is disabled, the results of multiple executions may be different. This is generally caused by asynchronous multi-thread executions during operator implementation, which changes the accumulation sequence of floating-point numbers.

However, if a model produces inconsistent execution results across multiple runs or requires accuracy optimization, you can enable deterministic computing to assist with model debugging and tuning. Note that if you want a completely definite result, you need to set a definite random seed in the training script to ensure that the random numbers generated in the program are also definite.

Example:

custom_op.parameter_map["deterministic"].i = 1

Training/Online inference

atomic_clean_policy

Example:

custom_op.parameter_map["atomic_clean_policy"].i = 1

Training/Online inference

static_memory_policy

Example:

custom_op.parameter_map["static_memory_policy"].i = 0

Training/Online inference

variable_use_1g_huge_page

In recommendation models, the embedding layer in TensorFlow uses variables. When embedding layers serve as input or output addresses for index-based operators (such as Gather and ScatterNd), large memory footprints may lead to extensive scattered access, potentially causing performance degradation. In such cases, you can try configuring this parameter to allocate memory for variables and constants using 1 GB huge pages, thereby improving memory access performance.

Using 1 GB huge pages can effectively reduce the number of page table entries and expand the address range covered by the translation lookaside buffer (TLB) cache, thereby improving performance for scattered access patterns. The TLB is a hardware module on the Ascend AI processor that caches recently used virtual-to-physical address mappings.

custom_op.parameter_map["variable_use_1g_huge_page"].i = 2

Training/Online inference

external_weight

When multiple models are loaded in a session, if the weights of these models can be reused, you are advised to use this configuration item to externalize the weights of the Const/Constant nodes on the network to implement weight reuse among multiple models and reduce the memory usage of the weights.

• False (default): The weights are not externalized but are saved in graphs.
• True: The weights of all Const/Constant nodes on the network are flushed to the disk and are converted to the FileConstant type. The weight file is named in the format of weight_<hash value>.

  If the environment variable ASCEND_WORK_PATH is not configured in the environment, the weight files are flushed to the current execution directory tmp_weight_<pid>_<sessionid>.

  If ASCEND_WORK_PATH is configured in the environment, the weight files are flushed to the ${ASCEND_WORK_PATH}/tmp_weight_<pid>_<sessionid> directory. For details about ASCEND_WORK_PATH, see "Installation" in Environment Variables.

  When the model is uninstalled, the tmp_weight_<pid>_<sessionid> directory is automatically deleted.

Note: This parameter is usually not required. If the model loading environment has limitations on memory, you can flush the weight externally.

custom_op.parameter_map["external_weight"].b = True

Training/Online inference

input_fusion_size

Note: This parameter takes effect only for static shape graphs.

Example:

custom_op.parameter_map["input_fusion_size"].i = 25600

Training/Online inference

input_batch_cpy

Example:

custom_op.parameter_map["input_batch_cpy"].b = True

Training/Online inference

input_shape

Input shape.

Configuration example:

custom_op.parameter_map["input_shape"].s = tf.compat.as_bytes("data:1,1,40,-1;label:1,-1;mask:-1,-1")

In the preceding example, the network model has three inputs: data (1, 1, 40, –1), label (1, –1), and mask (–1, –1). Separate the name and shapes of each input with colons (:). –1 indicates a dynamic dimension, whose size profiles are configured by using dynamic_dims.

Notes:

• The names entered in input_shape must be in the same alphabetical order as the names of the actual data nodes. For example, for inputs data, label, and mask, the names entered in input_shape must be in the order of data, label, and mask.

• If a network has both dataset inputs and placeholder inputs, since dynamic inputs are described in only one of the modes, you only need to fill in the shapes of the dynamic inputs.
• For scalar inputs, set the shape to 0.
• The shape range specified by this option must be valid.

Online inference

dynamic_dims

Input dimension size choices. Separate the dimension sizes by a semicolon (;). The dimension values match to the –1 placeholders in the input_shape argument with ordering preserved, and the number of –1 placeholders equals the number of dimension sizes of each profile. Set at least two dynamic dimension size profiles.

The argument of dynamic_dims must match that of input_shape, as failure to do so may lead to an error and system's exit.

Example:

custom_op.parameter_map["dynamic_dims"].s = tf.compat.as_bytes("20,20,1,1;40,40,2,2;80,60,4,4")

Based on the input_shape information in the preceding example, the supported input shape profiles are as follows:

• Profile 0: data(1,1,40,20)+label(1,20)+mask(1,1)
• Profile 1: data(1,1,40,40), label(1,40), mask(2,2)
• Profile 2: data(1,1,40,80)+label(1,60)+mask(4,4)

Notes:

Online inference

dynamic_node_type

Type of the dynamic input node.

• 0: dataset input
• 1: placeholder input

Only one type of dynamic inputs is allowed, dataset or placeholder.

custom_op.parameter_map["dynamic_node_type"].i = 0

Online inference

compile_hybrid_mode

Configuration example:

custom_op.parameter_map["compile_hybrid_mode"].i = 1

Online inference

ac_parallel_enable

Whether to allow AI CPU operators and AI Core operators to run in parallel in a dynamic shape graph.

Configuration example:

custom_op.parameter_map["ac_parallel_enable"].s = tf.compat.as_bytes("1")

Training/Online inference

compile_dynamic_mode

Configuration example:

custom_op.parameter_map["compile_dynamic_mode"].b = True

Note: This option cannot be used together with parameters for dynamic dimension size profiles (input_shape, dynamic_dims, and dynamic_node_type).

Training/Online inference

all_tensor_not_empty

Configuration example:

custom_op.parameter_map["all_tensor_not_empty"].b = True

Training/Online inference

mix_compile_mode

Mixed computing

• True: enabled.
• False (default): disabled.

In full offload mode, all compute operators are offloaded to the device. As a supplement to the full offload mode, mixed computing allows certain operators to be executed online within the frontend framework, improving the Ascend AI Processor's adaptability to TensorFlow.

Example:

custom_op.parameter_map["mix_compile_mode"].b =  True

Training/Online inference

in_out_pair_flag

Whether to offload operators specified by in_out_pair to the Ascend AI Processor in mixed computing scenarios.

• True (default)
• False

Example:

custom_op.parameter_map['in_out_pair_flag'].b = False

Online inference

in_out_pair

Names of the input-layer and output-layer operators offloaded (or not) in mixed computing scenarios.

Note that this option supports only one operator configured within the range of [in_nodes, out_nodes].

Example:

# Enable mixed computing.
custom_op.parameter_map["mix_compile_mode"].b = True
# Offload operators within the [in_nodes, out_nodes] range to the Ascend AI Processor for execution, and execute other operators in the frontend framework.
in_nodes.append('import/conv2d_1/convolution')
out_nodes.append('import/conv2d_59/BiasAdd')
out_nodes.append('import/conv2d_67/BiasAdd')
out_nodes.append('import/conv2d_75/BiasAdd')
all_graph_iop.append([in_nodes, out_nodes])
custom_op.parameter_map['in_out_pair'].s = tf.compat.as_bytes(str(all_graph_iop))
# Alternatively, retain operators within the [in_nodes, out_nodes] range for execution in the frontend framework, and offload other operators to the Ascend AI Processor for execution.
in_nodes.append('import/conv2d_1/convolution')
out_nodes.append('import/conv2d_59/BiasAdd')
out_nodes.append('import/conv2d_67/BiasAdd')
out_nodes.append('import/conv2d_75/BiasAdd')
all_graph_iop.append([in_nodes, out_nodes])
custom_op.parameter_map['in_out_pair_flag'].b = False
custom_op.parameter_map['in_out_pair'].s = tf.compat.as_bytes(str(all_graph_iop))

Online inference

enable_exception_dump

For details about the environment variable, see Environment Variables.

custom_op.parameter_map["enable_exception_dump"].i = 1

Training/Online inference

op_debug_config

Enable global memory check.

The value is the path of the .cfg configuration file. Multiple options in the configuration file are separated by commas (,).

• oom: checks whether memory overwriting occurs in the global memory during operator execution.

  During operator compilation, the .o file (operator binary file) and .json file (operator description file) are retained in the kernel_meta folder in the current execution path, and the following detection logic is added:

  inline __aicore__ void  CheckInvalidAccessOfDDR(xxx) {
      if (access_offset < 0 || access_offset + access_extent > ddr_size) {
          if (read_or_write == 1) {
              trap(0X5A5A0001);
          } else {
              trap(0X5A5A0002);
          }
      }
  }

  You can use dump_cce to view the preceding code in the generated .cce file.

  If memory overwriting occurs during compilation, the error code EZ9999 is reported.

• dump_cce: Retains the .cce file, .o file, and .json file of the operator in the kernel_meta folder in the current execution path during operator compilation.
• dump_loc: Retains the .cce file, .o file, and .json file of the operator, as well as the _loc.json file (mapping file of python-cce) in the kernel_meta folder in the current execution path during operator compilation.
• ccec_O0: Enables the default option -O0 of the CCEC during operator compilation. This option does not perform any optimization based on the debugging information.
• ccec_g: Enables the -g option of the CCEC during operator compilation. Compared with -O0, this option generates optimization and debugging information.
• check_flag: Checks whether the pipeline synchronization signals in an operator are valid and consistent during operator execution.

  Retain the .o file and .json file in the generated kernel_meta folder and add the following detection logic during operator compilation:

    set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
    set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
    set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
    set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
    ....
    pipe_barrier(PIPE_MTE3);
    pipe_barrier(PIPE_MTE2);
    pipe_barrier(PIPE_M);
    pipe_barrier(PIPE_V);
    pipe_barrier(PIPE_MTE1);
    pipe_barrier(PIPE_ALL);
    wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
    wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
    wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
    wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
    ...

  You can use dump_cce to view the preceding code in the generated .cce file.

  During compilation, if a mismatch exists in the pipeline synchronization signals in an operator, a timeout error is reported at the faulty operator. The following is an example of the error message:

  Aicore kernel execute failed, ..., fault kernel_name=Operator name,...
  rtStreamSynchronizeWithTimeout execute failed....

Example:

custom_op.parameter_map["op_debug_config"].s = tf.compat.as_bytes("/root/test0.cfg")

The information about the test0.cfg file is as follows:

op_debug_config=ccec_g,oom

Restrictions

The following is an example of the test0.cfg file:

op_debug_config= ccec_g,oom
op_debug_list=GatherV2,opType::ReduceSum

During model compilation, the GatherV2 and ReduceSum operators are compiled based on the ccec_g and oom options.

Training/Online inference

debug_dir

Directory of the debug files generated during operator building, including the .o, .json, and .cce files.

The path priority for storing the debugging files generated during operator compilation is as follows:

debug_dir > ASCEND_WORK_PATH > default storage path (current script execution path).

For details about the environment variable ASCEND_WORK_PATH, see Environment Variables.

Example:

custom_op.parameter_map["debug_dir"].s = tf.compat.as_bytes("/home/test")

Training/Online inference

export_compile_stat

Example:

custom_op.parameter_map["export_compile_stat"].i = 1

Training/Online inference

precision_mode_v2

Operator precision mode, which must be of the string type.

• fp16

  Indicates that float16 is forcibly selected if the operator precision in the original graph is float16, bfloat16, or float32.

• origin
  Retains the original precision.

  • If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only float32 and bfloat16, the system automatically uses high-precision float32.
  • If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only bfloat16, the AI CPU operator of float16 is used. If the AI CPU operator is not supported, an error is reported.
  • If the precision of an operator in the original graph is float32, and the implementation of the operator in the AI Core does not support float32 but supports only float16, the AI CPU operator of float32 is used. If the AI CPU operator is not supported, an error is reported.
• cube_fp16in_fp32out
  The system selects a processing mode based on the operator type for AI Core operators supporting both float32 and float16.

  • For cube operators, the system processes the computation based on the operator implementation.
    1. The preferred input data type is float16 and the output data type is float32.
    2. If the float16 input data and float32 output data types are not supported, set both the input and output data types to float32.
    3. If the float32 input and output data types are not supported, set both the input and output data types to float16.
    4. If the float16 input and output data types are not supported, an error is reported.
  • For vector compute operators, the operator precision in the original graph is float16 or bfloat16, and float32 is forcibly selected.

    This option is invalid if the original graph contains operators not supporting float32 in the AI Core, for example, an operator that supports only float16. In this case, float16 is retained. If the operator in the AI Core does not support float32 and it is configured to the blocklist of precision reduction (by setting precision_reduce to false), the counterpart AI CPU operator supporting float32 is used. If the AI CPU operator does not support float32, an error is reported.

• mixed_float16

  Mixed precision of float16, bfloat16, and float32 is used for neural network processing. For float32 and befloat16 operators in the original graph, float16 is automatically used for certain float32 and bfloat16 operators based on the built-in tuning policy. This will improve system performance and reduce memory usage with minimal precision degradation.

  Use the mixed precision mode in conjunction with loss scaling to compensate for the accuracy degradation caused by precision reduction.

• mixed_bfloat16

  Mixed precision of bfloat16 and float32 is used for neural network processing. In this mode, bfloat16 is automatically used for certain float32 operators in the original graph based on the built-in tuning policy. This will improve system performance and reduce memory usage with minimal precision degradation. If the operators do not support bfloat16 and float32, the AI CPU operators are used for computation. If AI CPU operators also do not support float16 and float32, an error is reported during execution.

  Note: This configuration is supported only by Atlas A3 training products / Atlas A3 inference products Atlas A2 training products / Atlas A2 inference products .

In online inference scenarios, the default value is fp16.

Example:

custom_op.parameter_map["precision_mode_v2"].s = tf.compat.as_bytes("origin")

Training/Online inference

precision_mode

Operator precision mode, which must be of the string type.

• allow_fp32_to_fp16
  • For matrix operators:
    • If the operator precision in the original graph is float32, the precision is preferably reduced to float16. If the operator in the AI Core does not support float16, float32 is used. If the operator in the AI Core does not support float32, the AI CPU operator is used for computation. If the AI CPU operator also does not support float32, an error is reported during execution.
    • If the operator precision in the original graph is bfloat16, the precision of the original graph is preferably used. If the operator in the AI Core does not support bfloat16, float32 is used. If the operator in the AI Core does not support float32, the precision is directly reduced to float16. If the operator in the AI Core does not support float16, the AI CPU operator is used for computation. If the AI CPU operator also does not support float16, an error is reported during execution.
  • For vector operators, the precision of the original graph is retained preferably.
    • If the operator precision in the original graph is float32, the precision of the original graph is preferably used. If the operator in the AI Core does not support float32, the precision is directly reduced to float16. If the operator in the AI Core does not support float16, the AI CPU operator is used for computation. If the AI CPU operator also does not support float16, an error is reported during execution.
    • If the operator precision in the original graph is bfloat16, the precision of the original graph is preferably used. If the operator in the AI Core does not support bfloat16, float32 is used. If the operator in the AI Core does not support float32, the precision is directly reduced to float16. If the operator in the AI Core does not support float16, the AI CPU operator is used for computation. If the AI CPU operator also does not support float16, an error is reported during execution.
• force_fp16

  Forces float16 for operators supporting float16, bfloat16, and float32. This parameter applies only to online inference scenarios.

• force_fp32/cube_fp16in_fp32out
  force_fp32 and cube_fp16in_fp32out have the same effect. This option indicates that the system selects different processing modes based on the operator type when the operator in the AI Core supports both the float32 and float16 data types. cube_fp16in_fp32out is newly added to the new version. For cube operators, this option has clearer semantics.

  • For cube operators, the system processes the computation based on the operator implementation.
    1. The preferred input data type is float16 and the output data type is float32.
    2. If the float16 input data and float32 output data types are not supported, set both the input and output data types to float32.
    3. If the float32 input and output data types are not supported, set both the input and output data types to float16.
    4. If the float16 input and output data types are not supported, an error is reported.
  • For vector compute operators, the operator precision in the original graph is float16 or bfloat16, and float32 is forcibly selected.

    This option is invalid if the original graph contains operators not supporting float32 in the AI Core, for example, an operator that supports only float16. In this case, float16 is retained. If the operator in the AI Core does not support float32 and it is configured to the blocklist of precision reduction (by setting precision_reduce to false), the counterpart AI CPU operator supporting float32 is used. If the AI CPU operator does not support float32, an error is reported.

• must_keep_origin_dtype
  Retains the original precision.

  • If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only float32 and bfloat16, the system automatically uses high-precision float32.
  • If the precision of an operator in the original graph is float16, and the implementation of the operator in the AI Core does not support float16 but supports only bfloat16, the AI CPU operator of float16 is used. If the AI CPU operator is not supported, an error is reported.
  • If the precision of an operator in the original graph is float32, and the implementation of the operator in the AI Core does not support float32 but supports only float16, the AI CPU operator of float32 is used. If the AI CPU operator is not supported, an error is reported.
• allow_mix_precision_fp16/allow_mix_precision

  allow_mix_precision has the same effect as that of allow_mix_precision_fp16, indicating that mixed precision of float16, bfloat16, and float32 is used for neural network processing. allow_mix_precision_fp16 is newly added to the new version, which has clearer semantics for easy understanding.

  For float32 and befloat16 operators in the original model, float16 is automatically used for certain float32 and bfloat16 operators based on the built-in tuning policy. This will improve system performance and reduce memory usage with minimal precision degradation.

• allow_mix_precision_bf16

  Mixed precision of bfloat16 and float32 is used for neural network processing. In this mode, bfloat16 is automatically used for certain float32 operators on the original model based on the built-in tuning policy. This will improve system performance and reduce memory usage with minimal precision degradation. If the operator in the AI Core does not support bfloat16 and float32, the AI CPU operator is used for computation. If AI CPU operator also does not support bfloat16 and float32, an error is reported during execution.

  Note: This configuration is supported only by Atlas A3 training products / Atlas A3 inference products Atlas A2 training products / Atlas A2 inference products .

• allow_fp32_to_bf16
  • If the operator precision in the original graph is float32, the precision of the original graph is preferably used. If the operator in the AI Core does not support float32, the precision is reduced to bfloat16. If the operator in the AI Core does not support bfloat16, the AI CPU operator is used for computation. If the AI CPU operator also does not support bfloat16, an error is reported during execution.
  • If the operator precision in the original graph is bfloat16, the precision of the original graph is preferably used. If the operator in the AI Core does not support bfloat16, float32 is used. If the operator in the AI Core does not support float32, the AI CPU operator is used for computation. If the AI CPU operator also does not support float32, an error is reported during execution.

  Note: This configuration is supported by Atlas A3 training products / Atlas A3 inference products Atlas A2 training products / Atlas A2 inference products .

In online inference scenarios, the default value is force_fp16.

Example:

custom_op.parameter_map["precision_mode"].s = tf.compat.as_bytes("allow_mix_precision")

Training/Online inference

modify_mixlist

When mixed precision is enabled, you can use this parameter to specify the path and file name of the blocklist, trustlist, and graylist, and specify the operators that allow precision reduction and those that do not allow precision reduction.

You can enable the mixed precision by configuring precision_mode_v2 (recommended) or precision_mode in the script.

custom_op.parameter_map["modify_mixlist"].s = tf.compat.as_bytes("/home/test/ops_info.json")

You can specify the operator types in ops_info.json as follows. Separate operators with commas (,).

{
  "black-list": {                  // Blocklist
     "to-remove": [                // Move an operator from the blocklist to the graylist.
     "Xlog1py"
     ],
     "to-add": [                   // Move an operator from the trustlist or graylist to the blocklist.
     "MatMul",
     "Cast"
     ]
  },
  "white-list": {                  // Trustlist
     "to-remove": [                // Move an operator from the trustlist to the graylist.
     "Conv2D"
     ],
     "to-add": [                   // Move an operator from the blocklist or graylist to the trustlist.
     "Bias"
     ]
  }
}

Note: The operators in the preceding example configuration file are for reference only. The configuration should be based on the actual hardware environment and the built-in tuning policies of the operators.

You can query the built-in tuning policy of each operator in mixed precision mode in CANN software installation directory/opp/built-in/op_impl/ai_core/tbe/config/<soc_version>/aic-<soc_version>-ops-info-<opType>.json. Example:

"Conv2D":{
    "precision_reduce":{
        "flag":"true"
},
...
}

• true (trustlist): The precision of operators on the trustlist can be reduced in mixed precision mode.
• false (blocklist): The precision of operators on the blocklist cannot be reduced in mixed precision mode.
• Not specified (graylist): Follows the same mixed precision processing as the upstream operator.

Training/Online inference

customize_dtypes

If precision_mode_v2 or precision_mode is used to set the global precision mode of a network, precision problems may occur on particular operators. In this case, you can use customize_dtypes to configure the precision mode of these operators, and still compile other operators using the precision mode specified by precision_mode_v2 or precision_mode. Note if precision_mode_v2 is set to origin or precision_mode is set to must_keep_origin_dtype, customize_dtypes does not take effect.

Set it to the path (including the name of the configuration file), for example, /home/test/customize_dtypes.cfg.

Configuration example:

custom_op.parameter_map["customize_dtypes"].s = tf.compat.as_bytes("/home/test/customize_dtypes.cfg")

List the names or types of operators whose precision needs customization in the configuration file. Each operator occupies a line, and the operator type must be defined based on Ascend IR. If both operator name and type are configured for an operator, the operator name applies during building.

The structure of the configuration file is as follows:

# By operator name
Opname1::InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
Opname2::InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
# By operator type
OpType::TypeName1:InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...
OpType::TypeName2:InputDtype:dtype1,dtype2,...OutputDtype:dtype1,...

Example:

# By operator name
resnet_v1_50/block1/unit_3/bottleneck_v1/Relu::InputDtype:float16,int8,OutputDtype:float16,int8
# By operator type
OpType::Relu:InputDtype:float16,int8,OutputDtype:float16,int8

Online inference/Training

enable_dump

Data dump enable.

• True: enabled. The dump file path is read from dump_path. If dump_path is set to None, an exception occurs.
• False (default): disabled.

custom_op.parameter_map["enable_dump"].b = True

Training/Online inference

dump_mode

Dump mode. The values are as follows:

• input: dumps only operator inputs.
• output (default): dumps only operator outputs.
• all: dumps both operator inputs and outputs.

Example:

custom_op.parameter_map["dump_mode"].s = tf.compat.as_bytes("all") 

Training/Online inference

enable_dump_debug

Example:

custom_op.parameter_map["enable_dump_debug"].b = True

Training

dump_debug_mode

Example:

custom_op.parameter_map["dump_debug_mode"].s = tf.compat.as_bytes("all") 

Training

dump_path

Dump path. This option is required when enable_dump or enable_dump_debug is set to True.

Create the specified path in advance in the environment (either container or host) where training is performed. The running user configured during installation must have the read and write permissions on this path. The path can be an absolute path or a path relative to the path where the training script is executed.

• An absolute path starting with a slash (/), for example, /home/test/output.
• A relative path starting with a directory name, for example, output.

Example:

custom_op.parameter_map["dump_path"].s = tf.compat.as_bytes("/home/test/output")  

Training/Online inference

dump_step

Iterations to dump.

Separate multiple iterations using vertical bars (|), for example, 0|5|10. You can also use hyphens (-) to specify the iteration range, for example, 0|3-5|10.

If this option is not set, dump data of all iterations is collected.

Example:

custom_op.parameter_map["dump_step"].s = tf.compat.as_bytes("0|5|10")

Training

dump_data

Type of operator content to dump.

• tensor (default): dumps operator data.
• stats: dumps operator statistics. The result file is in .csv format.

In large-scale training scenarios, dumping a large amount of data takes a long time. You can dump the statistics of all operators, identify the operators that may be abnormal based on the statistics, and then dump the input or output data of these abnormal operators.

Example:

custom_op.parameter_map["dump_data"].s = tf.compat.as_bytes("stats") 

Training/Online inference

dump_layer

Name of the operator to dump. Multiple operator names are separated by spaces. If this parameter is not set, all operators are dumped by default.

If the input of the specified operator involves the data operator, the data operator information is also dumped.

Example:

custom_op.parameter_map["dump_layer"].s = tf.compat.as_bytes("nodename1 nodename2 nodename3") 

Training/Online inference

quant_dumpable

Example:

custom_op.parameter_map["quant_dumpable"].s = tf.compat.as_bytes("1") 

Online inference

fusion_switch_file

Directory of the fusion switch configuration file, including the file name.

The value can contain letters, digits, underscores (_), hyphens (-), and periods (.).

The built-in graph fusion and UB fusion patterns are enabled by default. You can disable selected fusion patterns in the configuration file as needed. For details about fusion patterns that can be disabled, see Graph Fusion and UB Fusion Patterns.

Example:

custom_op.parameter_map["fusion_switch_file"].s = tf.compat.as_bytes("/home/test/fusion_switch.cfg")

The following is a template of the fusion_switch.cfg configuration file. on indicates that a fusion pattern is enabled, and off indicates that a fusion pattern is disabled.

{
    "Switch":{
        "GraphFusion":{
            "RequantFusionPass":"on",
            "ConvToFullyConnectionFusionPass":"off",
            "SoftmaxFusionPass":"on",
            "NotRequantFusionPass":"on",
            "SplitConvConcatFusionPass":"on",
            "ConvConcatFusionPass":"on",
            "MatMulBiasAddFusionPass":"on",
            "PoolingFusionPass":"on",
            "ZConcatv2dFusionPass":"on",
            "ZConcatExt2FusionPass":"on",
            "TfMergeSubFusionPass":"on"
        },
        "UBFusion":{
            "TbePool2dQuantFusionPass":"on"
        }
    }
}

To disable all fusion patterns at a time, refer to this configuration file example.

{
    "Switch":{
        "GraphFusion":{
            "ALL":"off"
        },
        "UBFusion":{
            "ALL":"off"
         }
    }
}

Notes:

1. Some built-in fusion patterns are not switchable due to functionality restrictions and these fusion patterns will remain enabled despite user's switch settings.
2. To disable all fusion patterns except selected ones, refer to the following example.

   {
       "Switch":{
           "GraphFusion":{
               "ALL":"off",
               "SoftmaxFusionPass":"on"
           },
           "UBFusion":{
               "ALL":"off",
               "TbePool2dQuantFusionPass":"on"
           }
       }
   }

Training/Online inference

buffer_optimize

Enables buffer optimization. This is an advanced switch.

• l2_optimize (default): enabled
• off_optimize: disabled.

Example:

custom_op.parameter_map["buffer_optimize"].s = tf.compat.as_bytes("l2_optimize")

Online inference

use_off_line

Enable training on the Ascend AI Processor.

• True (default): enabled.
• False: disabled. Training is performed on the host CPU.

Example:

custom_op.parameter_map["use_off_line"].b = True

Training/Online inference

profiling_mode

Enables profiling.

• True: enabled. The profiling options are determined by profiling_options.
• False (default): disabled.

custom_op.parameter_map["profiling_mode"].b = True

Note: The priority of this configuration item is higher than that of the environment variable PROFILING_MODE. For details about the environment variable, see "Profile Data Collection" in Environment Variables.

Training/Online inference

profiling_options

Sets profiling options.

• output: path for storing profiling result files. Both absolute path and relative path (relative to the path where the command is run) are supported. The path cannot contain the following special characters: "\n", "\f", "\r", "\b", "\t", "\v", and "\u007F".
  • An absolute path starts with a slash (/), for example, /home/output.
  • A relative path starts with a directory name, for example, output.
  • It takes precedence over ASCEND_WORK_PATH.
  • This path does not need to be created in advance because it is automatically created during collection.
• storage_limit: maximum size of files that can be stored in a specified disk directory. If the size of profile data files in the disk is about to use up the maximum storage space specified by this option or the total remaining disk space is about to be used up (remaining space ≤ 20 MB), the earliest files in the disk are aged and deleted.

  The value range is [200, 4294967295], and the unit is MB. The unit must be included when you set this parameter, for example, 200 MB.

  If this parameter is not set, the default value is 90% of the available space of the disk where the directory for storing profile data files is located.

• training_trace: iteration tracing switch. Collects software profile data of a training job and the AI Software Stack to profile the training job. Focuses on forward and backward propagation, and gradient aggregation and update. This option must be set to on when the forward and backward propagation operator data is collected.
• task_trace and task_time: switches that control collection of the operator delivery and execution durations. Related duration data must be output to the task_time, op_summary, and op_statistic files. Possible configuration values are as follows:
  • on: switch on. This is the default value, delivering the same effect as l1.
  • off: switch off.
  • l0: collects operator delivery and execution duration data. Compared with l1, l0 does not collect basic operator information, so the performance overhead during collection is smaller, and this enables more accurate collection of statistics on time duration data.
  • l1: collects operator delivery and execution duration data, as well as basic operator information, to provide more comprehensive performance analysis data.

  When Profiling is enabled to collect training data, task_trace and training_trace must be set to on.

• ge_api: switch that controls collection of the time consumption data of dynamic-shape operators in the host scheduling phase. Possible values are:
  • off: switch off. The default value is off.
  • l0: collects the time consumption data of dynamic-shape operators in the main host scheduling phase to facilitate accurate statistics.
  • l1: collects finer-grained time consumption data of dynamic-shape operators in the host scheduling phase to provide more comprehensive performance analysis data.
• hccl: communication data collection switch, either on or off (default).
  NOTE:

  This switch will be deprecated in later versions. To control data collection, use task_time.

• aicpu: whether to collect details about the AI CPU operator, such as the operator execution time and data copy time. The value can be on or off (default).
• fp_point: start point of the forward propagated operator in iteration traces, to record the start timestamp of forward propagation. Set the value to the name of the top operator in forward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain the name. Alternatively, you can leave this option empty (for example, "fp_point":""), and the system will automatically identify the start point of the forward propagated operator.
• bp_point: end point of the backward propagated operator in iteration traces, to record the end timestamp of backward propagation. bp_point and fp_point are used to compute the time used by forward and backward propagation. Set the value to the name of the bottom operator in backward propagation. You can save the graph as a .pbtxt file by using tf.io.write_graph in the training script to obtain the name. Alternatively, you can leave this option empty (for example, "bp_point":""), and the system will automatically identify the end point of the backward propagated operator.
• aic_metrics: AI Core metric to profile. The options are as follows:
  • ArithmeticUtilization: arithmetic utilization ratio.
  • PipeUtilization (default): ratio of time taken by the compute units to that of MTEs.
  • Memory: ratio of external memory read/write instructions.
  • MemoryL0: ratio of internal memory L0 read/write instructions.
  • MemoryUB: ratio of internal memory UB read/write instructions.
  • ResourceConflictRatio: ratio of pipeline queue instructions.
  • L2Cache: read/write L2 cache hits and re-allocations after cache misses

    Atlas inference products : This parameter is not supported.

    Atlas training products : This parameter is not supported.

  • MemoryAccess: bandwidth of the operator's memory access on cores.

    Atlas inference products : This parameter is not supported.

    Atlas training products : This parameter is not supported.

  NOTE:

  The registers whose data is to be collected can be customized, for example, "aic_metrics":"Custom:0x49,0x8,0x15,0x1b,0x64,0x10".

  • The Custom field indicates the customization type. It is set to specific register values in the range of [0x1, 0x6E].
  • A maximum of eight registers can be configured, which are separated with commas (,).
  • The register value can be in hexadecimal or decimal format.
• l2: switch that controls L2 cache and TLB page table cache hit ratio, either on or off (default).
  • Atlas inference products : supports collection of the L2 cache hit ratio.
  • Atlas training products : supports collection of the L2 cache hit ratio.
  • Atlas A2 training products / Atlas A2 inference products : supports collection of L2 cache and TLB page table cache hit ratio. aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 cache from the AI Core.
  • Atlas A3 training products / Atlas A3 inference products : supports collection of L2 cache and TLB page table cache hit ratio. aic-metrics=L2Cache is recommended for analyzing the number of hits on L2 cache from the AI Core.
• msproftx: switch that controls the msproftx user and upper-layer framework program to output profile data, either on or off (default).

  Add the following mstx API or msproftx API to the application script. The mstx API is recommended.

• runtime_api: runtime API data collection switch, either on or off (default). You can collect runtime API profile data, including the synchronous/asynchronous memory replication latencies between the host and device and between devices.
• sys_hardware_mem_freq: switch that controls the collection of the on-chip memory, QoS transmission bandwidth, LLC L3 cache bandwidth, accelerator bandwidth, SoC transmission bandwidth, and component memory usage. The collected content varies depending on the product. The actual result prevails. The value range is [1, 100], in Hz.

  Sampling memory data in the environment where glibc (2.34 or earlier) is installed may trigger a known Bug 19329. This problem can be solved by upgrading the glibc version.

  NOTE:

  For the following products, you are advised not to increase the profiling frequency after the profiling task is complete. Otherwise, SoC transmission bandwidth data may be lost.

  Atlas 200I/500 A2 inference products

  Atlas A2 training products / Atlas A2 inference products

  Atlas A3 training products / Atlas A3 inference products

• llc_profiling: LLC events to profile. Possible values are as follows:
  • read (default): read events, that is, the L3 cache read rate.
  • write: write events, that is, the L3 cache write rate.
• sys_io_sampling_freq: NIC and ROCE data collection frequency. The value range is [1, 100], in Hz.

  Atlas inference products : This parameter is not supported.

  Atlas A2 training products / Atlas A2 inference products : supports NIC and RoCE collection.

  Atlas A3 training products / Atlas A3 inference products : supports NIC and RoCE collection.

• sys_interconnection_freq: frequency of collecting collective communication bandwidth data (HCCS), SIO data, PCIe data and inter-chip transmission bandwidth information. The value range is [1, 50], in Hz.
  • Atlas training products : supports HCCS and PCIe data collection.
  • Atlas A2 training products / Atlas A2 inference products : supports HCCS, PCIe data, and inter-chip transmission bandwidth information collection.
  • Atlas A3 training products / Atlas A3 inference products : supports HCCS, PCIe data, inter-chip transmission bandwidth information, and SIO data collection.
• dvpp_freq: DVPP collection frequency. The value range is [1, 100], in Hz.
• instr_profiling: AI Core and AI Vector bandwidth and latency collection switch. The value can be on or off (default).
  • Atlas training products : This function is not supported.
  • Atlas A2 training products / Atlas A2 inference products : This switch is not supported. This function is controlled through instr_profiling_freq.
  • Atlas A3 training products / Atlas A3 inference products : This switch is not supported. This function is controlled through instr_profiling_freq.
• instr_profiling_freq: AI Core and AI Vector bandwidth and latency collection switch. If the collection frequency is configured, the related collection capability is enabled. The value range is [300, 30000]. The unit is Hz.
  • Atlas training products : This function is not supported.
  • Atlas A2 training products / Atlas A2 inference products : supported. However, instr_profiling_freq is mutually exclusive with training_trace, task_trace, hccl, aicpu, fp_point, bp_point, aic_metrics, l2, task_time, and runtime_api, so they cannot be executed at the same time.
  • Atlas A3 training products / Atlas A3 inference products : supported. However, instr_profiling_freq is mutually exclusive with training_trace, task_trace, hccl, aicpu, fp_point, bp_point, aic_metrics, l2, task_time, and runtime_api, so they cannot be executed at the same time.
• host_sys: switch for collecting host profile data. You can select one or more options and separate them with commas (,), for example, "host_sys": "cpu,mem".
  • cpu: process CPU utilization
  • mem: process memory utilization
• host_sys_usage: Host-side system and process CPU and memory data collection option, selected from cpu and mem. You can select one or more options and separate them with commas (,).
• host_sys_usage_freq: Host-side system and process CPU and memory data collection frequency. The value range is [1, 50] and the default value is 50. The unit is Hz.

Example:

custom_op.parameter_map["profiling_options"].s = tf.compat.as_bytes('{"output":"/tmp/profiling","training_trace":"on","fp_point":"","bp_point":""}')

Training/Online inference

aoe_mode

Tuning mode of AOE.

• 1: subgraph tuning.
• 2: operator tuning.
• 4: gradient splitting tuning.

  In the data parallelism scenario, AllReduce is used to aggregate gradients. The gradient splitting mode is closely related to the distributed training performance. If the splitting is improper, the communication hangover time is long after the backward propagation is complete, affecting the cluster training performance and linearity. It is sophisticated to perform manual tuning through the gradient splitting API (set_split_strategy_by_idx or set_split_strategy_by_size) of collective communication. The AOE tool collects profile data in the real-device environment and automatically looks up for the optimal splitting policy. You only need to set the obtained policy to your network by passing it to the set_split_strategy_by_idx call.

Example:

custom_op.parameter_map["aoe_mode"].s = tf.compat.as_bytes("2")

Training

work_path

Working directory of AOE, which stores the configuration and tuning result files. By default, the files are generated in the current directory.

The value is a string. Create the specified directory in advance in the environment (either container or host) where training is performed. The running user configured during installation must have the read and write permissions on this directory. The value can be an absolute path or a path relative to the path where the training script is executed.

• An absolute path starting with a slash (/), for example, /home/test/output.
• A relative path starting with a directory name, for example, output.

Example:

custom_op.parameter_map["work_path"].s = tf.compat.as_bytes("/home/test/output")

Training

aoe_config_file

Tunes only operators with low performance on the network with AOE. Set this parameter to the path and name of the configuration file that contains the operator information, for example, /home/test/cfg/tuning_config.cfg.

Example:

custom_op.parameter_map["aoe_config_file"].s=tf.compat.as_bytes("/home/test/cfg/tuning_config.cfg")

The configuration file contains information about the operators to be tuned. The file content format is as follows:

{
       "tune_ops_name":["bert/embeddings/addbert/embeddings/add_1","loss/MatMul"],
       "tune_ops_type":["Add", "Mul"],
       "tune_optimization_level":"O1",
       "feature":["deeper_opat"]
}

• tune_ops_name: name of the specified operator (whole word match). You can specify one or more operator names. If multiple operator names are specified, separate them with commas (,). The operator name must be the node name of the network model processed by Graph Compiler. You can obtain the operator name from profiling tuning data. For details, see Profiling Instructions.
• tune_ops_type: specified operator type (whole word match). You can specify one or more operator types. If multiple operator types are specified, separate them with commas (,). If a fused operator contains the specified operator type, the fused operator will also be tuned.
• tune_optimization_level: tuning mode. The value O1 indicates the high-performance tuning mode, and the value O2 indicates the normal mode. The default value is O2.
• feature: tuning feature switch. The value can be deeper_opat or nonhomo_split. The value deeper_opat indicates that in-depth operator tuning is enabled. In this case, aoe_mode must be set to 2. The value nonhomo_split indicates that non-uniform subgraph partition tuning is enabled. In this case, aoe_mode must be set to 1.

Training

op_compiler_cache_mode

Disk cache mode for operator building. enable is the default value.

• enable: disk cache mode enabled. The operator build information is cached to the disk, which can be reused by operators with the same build parameters, improving build efficiency.
• force: cache mode enabled. This mode deletes the existing cache, then recompiles the operators and adds them to the cache. For example, for Python changes, dependency library changes, or repository changes after operator optimization, you need to set this option to force to clean up the existing cache and then change it to enable to prevent the cache from being forcibly refreshed during each build. Note that you are not advised to set the force option for parallel program compilation. Otherwise, the cache used by other models may be cleaned up, causing compilation failures.
• disable: disabled.

Notes:

• When enabling the operator compilation cache function, you can configure the path for storing the operator compilation cache file by using op_compiler_cache_dir.
• disable and force are recommended for publishing the final model.
• If op_debug_level is set to a non-zero value, the op_compiler_cache_mode configuration is ignored, the operator compilation cache function is disabled, and all operators are recompiled.
• If op_debug_config is not empty and the op_debug_list field is not configured, the op_compiler_cache_mode configuration is ignored, the operator compilation cache function is disabled, and all operators are recompiled.
• If op_debug_config is not empty, the op_debug_list field is configured, and op_compiler_cache_mode is set to enable or force, the operators in the list are recompiled, and the operator compilation cache function is enabled for operators that are not in the list. However, operators that are not in the list will not be recompiled.
• When the operator compilation cache function is enabled, the default disk space allocated for cache files is 500 MB. If disk space becomes insufficient, cache files are deleted and 50% of the cache space is reserved by default. You can also customize the disk space allocated for cache files and the percentage of cache space to retain as follows:
  1. Using the op_cache.ini configuration file

     After the operator is compiled, the op_cache.ini file is automatically generated in the directory specified by op_compiler_cache_dir. You can use this file to set the disk space allocated for cache files and the percentage of cache space to retain. If the op_cache.ini file does not exist, manually create it.

     Add the following information to the op_cache.ini file:

     # Configure the file format (required). The automatically generated file contains the following information by default. When manually creating a file, enter the following information:
     [op_compiler_cache]
     # Limit the disk space of the cache folder on the Ascend AI Processor (unit: MB).
     max_op_cache_size=500
     # When the disk space is insufficient, set the percentage of cache files to retain. Value range: [1, 100] (%). For example, setting it to 80 means that when disk space becomes insufficient, 80% of the cache files will be retained and the rest will be deleted.
     remain_cache_size_ratio=80

     • The op_cache.ini file takes effect only when the values of max_op_cache_size and remain_cache_size_ratio in the preceding file are valid.
     • When the size of the compilation cache file exceeds the configured value of max_op_cache_size and the cache file has not been accessed for more than half an hour, the cache file will be aged out. (Operator compilation will not be interrupted if the cache file size exceeds the limit. Therefore, if max_op_cache_size is set too small, the actual compilation cache file size may exceed the configured value.)
     • To disable the compilation cache aging function, set max_op_cache_size to -1. In this case, the access time is not updated when the operator cache is accessed, the operator compilation cache is not aged, and the default disk space of 500 MB is used.
     • If multiple users use the same cache path, the configuration file affects all users.
  2. Using environment variable ASCEND_MAX_OP_CACHE_SIZE

     You can use the environment variable ASCEND_MAX_OP_CACHE_SIZE to limit the disk space for cache files under an Ascend AI Processor. When the compilation cache space reaches the value set by ASCEND_MAX_OP_CACHE_SIZE and a cache file has not been accessed for more than half an hour, the cache file will be aged out. ASCEND_REMAIN_CACHE_SIZE_RATIO can be used to set the percentage of cache space to retain. For details about environment variables, see "Operator Building" in Environment Variables.

     To disable the compilation cache aging function, set ASCEND_MAX_OP_CACHE_SIZE to -1.

  If both the op_cache.ini file and environment variables are configured, the configuration items in op_cache.ini take precedence. If neither is configured, the system uses the default values: 500 MB of disk space for the cache, with 50% of the cache space retained.

Example:

custom_op.parameter_map["op_compiler_cache_mode"].s = tf.compat.as_bytes("enable")

Training/Online inference

op_compiler_cache_dir

Disk cache directory for operator compilation.

The value can contain letters, digits, underscores (_), hyphens (-), and periods (.).

If the specified directory exists and is valid, the kernel_cache subdirectory is automatically created. If the specified directory does not exist but is valid, the system automatically creates a directory and the kernel_cache subdirectory.

The storage priority of operator compilation cache files is as follows:

op_compiler_cache_dir > ${ASCEND_CACHE_PATH}/kernel_cache > Default path ($HOME/atc_data)

For details about ASCEND_CACHE_PATH, see "Installation" in Environment Variables.

Example:

custom_op.parameter_map["op_compiler_cache_dir"].s = tf.compat.as_bytes("/home/test/kernel_cache")

Training/Online inference

aicore_num

Maximum number of Cube cores and Vector cores used for operator compilation.

Example:

custom_op.parameter_map["aicore_num"].s = tf.compat.as_bytes("2|4")

Training/Online inference

oo_constant_folding

Enables or disables constant folding.

Example:

custom_op.parameter_map["oo_constant_folding"].b = True

Training/Online inference

local_rank_id

Rank ID of the current process, used in data parallel processing. The main process deduplicates the data and distributes the deduplicated data to the devices of other processes for forward and backward propagation.

In this mode, multiple devices on a host share one process for data preprocessing. Although this is still a multi-process scenario, data preprocessing is performed in the main process, and other processes no longer accept datasets on the current process, but only receive preprocessed data from the main process.

To identify the main process, call the collective communication API get_local_rank_id() to get the rank ID of the current process on its server.

Example:

custom_op.parameter_map["local_rank_id"].i = 0

Training/Online inference

local_device_list

Devices that the main process sends data to, used in conjunction with local_rank_id.

custom_op.parameter_map["local_device_list"].s = tf.compat.as_bytes("0,1")

Training/Online inference

hccl_timeout

Synchronization timeout for inter-device task execution, in seconds.

You can set the timeout interval if the default value does not meet your requirement (for example, when a communication failure occurs).

• For Atlas training products , the value range is (0, 17340], in seconds. The default value is 1836.

  Note: For Atlas training products , actual timeout interval set in the system = (Value of this parameter // 68) × 68 (unit: s). If the parameter value is less than 68, 68s is used by default.

  For example, if hccl_timeout is set to 600, the actual timeout interval set in the system is 544s (600 // 68 × 68 = 8 × 68).

• For the Atlas 300I Duo inference card, the value range is (0, 17340], in seconds. The default value is 1836.

  Note: For Atlas 300I Duo inference card, actual timeout interval set in the system = (Value of this parameter // 68) × 68 (unit: s). If the parameter value is less than 68, 68s is used by default.

  For example, if hccl_timeout is set to 600, the actual timeout interval set in the system is 544s (600 // 68 × 68 = 8 × 68).

• For Atlas A2 training products / Atlas A2 inference products , the value range is [0, 2147483647], in seconds. The default value is 1836. The value 0 indicates that the session never times out.
• For Atlas A3 training products / Atlas A3 inference products , the value range is [0, 2147483647], in seconds. The default value is 1836. The value 0 indicates that the session never times out.

Example:

custom_op.parameter_map["hccl_timeout"].i = 1800

Training/Online inference

op_wait_timeout

Operator wait timeout interval (s). Defaults to 120.

You can set the timeout interval if the default value does not meet your requirement.

Configuration example:

custom_op.parameter_map["op_wait_timeout"].i = 120

Training/Online inference

op_execute_timeout

Operator execution timeout interval (s).

Example:

custom_op.parameter_map["op_execute_timeout"].i = 90

Training/Online inference

stream_sync_timeout

Timeout interval for stream synchronization during graph execution. If the timeout interval exceeds the configured value, a synchronization failure is reported. The unit is ms.

The default value is -1, indicating that there is no waiting time and no error is reported when the synchronization fails.

Note: In cluster scenarios, the value of this option (timeout interval for stream synchronization) must be greater than the collective communication timeout interval, that is, the value of hccl_timeout or the environment variable HCCL_EXEC_TIMEOUT.

Example:

custom_op.parameter_map["stream_sync_timeout"].i = 60000

Training/Online inference

event_sync_timeout

Timeout interval for event synchronization during graph execution. If the timeout interval exceeds the configured value, a synchronization failure is reported. The unit is ms.

The default value is -1, indicating that there is no waiting time and no error is reported when the synchronization fails.

Configuration example:

custom_op.parameter_map["event_sync_timeout"].i = 60000

Training/Online inference

graph_compiler_cache_dir

Drive cache directory for graph compilation. If this parameter is not empty, the drive cache function for graph compilation takes effect.

The graph compilation cache function supports drive persistence of graph compilation results. When graph compilation is performed again, the compilation results cached on the drive can be directly loaded to reduce the graph compilation duration.

Example:

custom_op.parameter_map["graph_compiler_cache_dir"].s = tf.compat.as_bytes("/root/build_cache_dir")

Training/Online inference

jit_compile

Determines whether to compile the operator online or use the compiled operator binary file.

• auto (default): For a static shape network, compile the operator online. For a dynamic shape network, search for the compiled operator binary file in the system first. If the corresponding binary file is not available, compile the operator.
• true: Operators are compiled online. The system performs fusion and tuning based on the obtained graph information to get better performing operators.
• false: The compiled operator binary file in the system is preferentially searched. If the file can be found, operators are not compiled anymore, which produces better compilation performance. If the file cannot be found, operators will be compiled.

Example:

custom_op.parameter_map["jit_compile"].s = tf.compat.as_bytes( "auto")

Training/Online inference

shape_generalization_mode

Example:

custom_op.parameter_map["shape_generalization_mode"].s = tf.compat.as_bytes( "FULL")

Training/Online inference

experimental_accelerate_train_mode

If training takes more than one hour, you can trigger training acceleration to improve training performance by configuring this option.

Based on the configured acceleration type, acceleration trigger mode, and the proportion of low-precision training processes, the software compiles and runs the corresponding proportion of training processes with reduced precision, while the remaining processes are compiled and run at their original precision.

Example:

• Step-triggered acceleration

  custom_op.parameter_map["experimental_accelerate_train_mode"].s =
  tf.compat.as_bytes("fast|step|0.9")

• Loss-triggered acceleration:

  custom_op.parameter_map["experimental_accelerate_train_mode"].s =
  tf.compat.as_bytes("fast|loss|1.05")

Notes:

1. To use this option for training acceleration, ensure that the network script can converge properly.
2. For training scripts with short execution time, enabling this option may not bring positive end-to-end performance gains.
3. The function of this option is related to the precision mode configured in the network script:
   • When precision_mode is used to configure the precision mode, this option can be enabled only when precision_mode is set to allow_fp32_to_fp16, must_keep_origin_dtype, or none.
   • When precision_mode_v2 is used to configure the precision mode, this option can be enabled only when precision_mode_v2 is set to origin or none.
4. This option is related to the number of small loops. Enabling small loops may result in inaccurate splitting of the training process based on the specified steps or loss value, which may ultimately affect the loss and accuracy.
5. When this option is enabled, you need to modify the network script based on the following rules:
   • If the entire training process is split by step, you need to set the STEP_NOW and TOTAL_STEP environment variables to notify the bottom layer of the step value and the total number of steps of each run.
   • If the entire training process is split by loss, you need to set the LOSS_NOW and TARGET_LOSS environment variables to notify the bottom layer of the loss value and target loss of each run.

   The following is an example of modifying the network script in step splitting mode:

   # Set the initial value of STEP_NOW to 0.
   os.environ['STEP_NOW'] =  "0"
   # Set TOTAL_STEP to the total number of steps.
   os.environ['TOTAL_STEP'] =  str(epoch)
   for i in range(epoch):
       # Start training.
       _, step = sess.run([train_op, global_step])
       # Update the value of STEP_NOW to the current step.
       os.environ['STEP_NOW'] =  str(step)

   The following is an example of modifying the network script in loss splitting mode:

   # Set the initial value of LOSS_NOW to the initial loss value of the network. The following value is for reference only.
   os.environ['LOSS_NOW'] =  "7.0"
   # Set the value of TARGET_LOSS to the target loss value. The following value is for reference only.
   os.environ['TARGET_LOSS'] =  "3.0"
   for i in range(epoch):
       # Start training.
       _, step = sess.run([train_op, global_step])
       # Update the value of LOSS_NOW to the loss value that is being executed.
       os.environ['LOSS_NOW'] =  str(loss)

Training

auto_multistream_parallel_mode

Training

op_debug_level

Function debugging.

Whether to enable operator debugging. The values are as follows:

• 0: disables operator debug.
• 1: Enables operator debug. TBE instruction mapping files are generated in the kernel_meta directory under the training script execution path, including operator CCE files (.cce), Python-CCE mapping files (_loc.json), .o files, and .json files. These files are used for AI Core error analysis with related tools.
• 2: Enables operator debug. TBE instruction mapping files are generated in the kernel_meta directory under the training script execution path, including operator CCE files (.cce), Python-CCE mapping files (_loc.json), .o files, and .json files. The compilation optimization of the CCE compiler is disabled and the CCE compiler debugging function is enabled (by setting the compiler option to -O0-g). These files are used for AI Core error analysis with related tools.
• 3: disables operator debug. The operator .o and .json files are retained in the kernel_meta folder in the training script execution directory.
• 4: disables operator debug. The operator binary (.o) and operator description file (.json) are retained, and a TBE instruction mapping file (.cce) and a UB fusion description file ({$kernel_name}_compute.json) are generated in the kernel_meta folder under the training script execution directory.
  NOTICE:

  • If this option is set to 0 and op_debug_config is configured, the operator compilation directory kernel_meta is still generated in the current execution path during training. The content generated in the directory is subject to op_debug_config.
  • You are advised to set this option to 0 or 3 for training. To locate AI Core errors, set this parameter to 1 or 2, which might compromise the network performance.
  • If this option is set to 2 (the CCE compiler is enabled), it cannot be used together with the oom option in op_debug_config. Otherwise, an AI Core error is reported. The following is an example of the error message:

    ...there is an aivec error exception, core id is 49, error code = 0x4 ...

  • If this parameter is set to 2 (the CCE compiler is enabled), the size of the operator kernel file (*.o file) increases. In dynamic shape scenarios, all possible scenarios are traversed during operator build, which may cause operator build failures due to large operator kernel files. In this case, 2 is not recommended.

    If the build failure is caused by the large operator kernel file, the following log is displayed:

    message:link error ld.lld: error: InputSection too large for range extension thunk ./kernel_meta_xxxxx.o:(xxxx)

  • If the value of this option is not 0, you can use the debug_dir option to specify the path for storing debugging-related process files.
  • If this option is set to 0 and NPU_COLLECT_PATH is set, the operator compilation directory kernel_meta is generated in the current path after the command is executed. If ASCEND_WORK_PATH is set, kernel_meta is generated in the path specified by the environment variable. For details about the environment variable, see Environment Variables.
  • When the debug function is enabled, if the model contains the following merged compute and communication (MC2) operators, the *.o, *.json, and *.cce files of the operators are not generated in the operator build folder kernel_meta.

    MatMulAllReduce

    MatMulAllReduceAddRmsNorm

    AllGatherMatMul

    MatMulReduceScatter

    AlltoAllAllGatherBatchMatMul

    BatchMatMulReduceScatterAlltoAll

This parameter is left empty by default, indicating that the configuration is disabled.

Example:

custom_op.parameter_map["op_debug_level"].i = 0

Training/Online inference

enable_data_pre_proc

Performance tuning.

Whether to enable GetNext operator offload to the Ascend AI Processor. GetNext operator offload is a prerequisite for iteration offload.

• True: enabled. The prerequisite for GetNext operator offload is that the TensorFlow Dataset mode is used to read data.
• False (default): disabled.

custom_op.parameter_map["enable_data_pre_proc"].b = True

Training

variable_format_optimize

Performance tuning.

Variable format optimization enable.

• True: enabled.
• False: disabled.

If it is enabled, the variables are reformatted during network variable initialization to better target to Ascend AI Processor (for example, from NCHW to NC1HWC0) for improved training efficiency. Enable or disable this function as needed.

This parameter is left empty by default, indicating that the configuration is disabled.

Example:

custom_op.parameter_map["variable_format_optimize"].b =  True

Training

op_select_implmode

Performance tuning.

Operator implementation mode select. Certain operators built in the Ascend AI Processor can be implemented in either high-precision or high-performance mode at model build time. Arguments:

• high_precision: high-precision implementation mode. In high-precision mode, Taylor's theorem or Newton's method is used to improve operator precision with float16 input.
• high_performance (default): high-performance implementation mode. In high-performance mode, the optimal performance is implemented without affecting the network precision (float16).

This parameter is left empty by default, indicating that the configuration is disabled.

Example:

custom_op.parameter_map["op_select_implmode"].s = tf.compat.as_bytes("high_precision")

Training/Online inference

optypelist_for_implmode

Performance tuning.

List of operator types (separated by commas) that use the mode specified by the op_select_implmode parameter. Currently, Pooling, SoftmaxV2, LRN, and ROIAlign operators are supported.

Use this option in conjunction with op_select_implmode, for example:

Set op_select_implmode to high_precision.

Set optypelist_for_implmode to Pooling.

This parameter is left empty by default, indicating that the configuration is disabled.

Example:

custom_op.parameter_map["optypelist_for_implmode"].s = tf.compat.as_bytes("Pooling,SoftmaxV2")

Training/Online inference

dynamic_input

Whether it is a dynamic input.

• True
• False (default)

Example:

custom_op.parameter_map["dynamic_input"].b = True

Training/Online inference

dynamic_graph_execute_mode

Execution mode of a dynamic input. That is, this option takes effect when dynamic_input is set to True. Possible values are:

dynamic_execute: dynamic graph compilation. In this mode, the shape range configured in dynamic_inputs_shape_range is used for compilation.

Example:

custom_op.parameter_map["dynamic_graph_execute_mode"].s = tf.compat.as_bytes("dynamic_execute")

Training/Online inference

dynamic_inputs_shape_range

Shape range of each dynamic input. If a graph has two dataset inputs and one placeholder input, a configuration example is as follows:

custom_op.parameter_map["dynamic_inputs_shape_range"].s = tf.compat.as_bytes("getnext:[128 ,3~5, 2~128, -1],[64 ,3~5, 2~128, -1];data:[128 ,3~5, 2~128, -1]")

Precautions:

• When this parameter is used, constants cannot be set to user inputs.

• getnext indicates the dataset inputs and data indicates the placeholder inputs.
• The size of a static dimension is specified by a determinant value. The size range of a dynamic dimension is specified by using a tilde (~). A dynamic dimension without size range specified is denoted by –1.
• Assume that your graph has three dataset inputs but the first dataset input has a static shape; the static shape must be specified as shown below.

  custom_op.parameter_map["dynamic_inputs_shape_range"].s = tf.compat.as_bytes("getnext:[3,3,4,10],[-1,3,2~1000,-1],[-1,-1,-1,-1]")

• For scalar inputs, you also need to fill in the shape range by using square brackets ([]). No space is allowed before the square brackets.
• If there are multiple getnext inputs or data inputs on the network, the input ordering must be preserved. For example:
  • If there are multiple dataset inputs on the network:

    def func(x):
        x = x + 1
        y = x + 2
        return x,y
    dataset = tf.data.Dataset.range(min_size, max_size)
    dataset = dataset.map(func)

    Assume that the first input of the network is x (shape range: [3–5]) and the second input is y (shape range: [3–6]). When configuring dynamic_inputs_shape_range, you must specify these shape ranges in the same order as the network inputs.

    custom_op.parameter_map["dynamic_inputs_shape_range"].s = tf.compat.as_bytes("getnext:[3~5],[3~6]")

  • If there are multiple placeholder inputs on the network:

    If the placeholder names are not specified, for example:

    x = tf.placeholder(tf.int32)
    y = tf.placeholder(tf.int32)

    The order of placeholders must match their definition order in the script. Assume that the first input of the network is x (shape range: [3~5]) and the second input is y (shape range: [3~6]). When configuring dynamic_inputs_shape_range, you must specify these shape ranges in the same order as the network inputs.

    custom_op.parameter_map["dynamic_inputs_shape_range"].s = tf.compat.as_bytes("data:[3~5],[3~6]")

    If the placeholder names are specified, for example:

    x = tf.placeholder(tf.int32, name='b')
    y = tf.placeholder(tf.int32, name='a')

    The network inputs are sorted in alphabetical order based on their names.

    For example, if the first input is y (shape range: [3~6]) and the second input is x (shape range: [3~5]), you must configure their shape ranges in dynamic_inputs_shape_range in the same order.

    custom_op.parameter_map["dynamic_inputs_shape_range"].s = tf.compat.as_bytes("data:[3~6],[3~5]")

  NOTICE:

  • For subgraphs with different input shapes, set_graph_exec_config is recommended for supporting dynamic inputs. dynamic_inputs_shape_range applies only to a single graph, which may cause execution errors.
  • If the placeholder names are not specified in the network script, the placeholders are named in the following format:

    xxx_0, xxx_1, xxx_2, ......

    The content following the underscore (_) is the sequence index of a placeholder in the network script. Placeholders are arranged in alphabetical order based on these indexes. If the number of placeholders is greater than 10, the sequence is xxx_0 -> xxx_10 -> xxx_2 -> xxx_3. In the network script, the placeholder with index 10 is placed before the placeholder with index 2. As a result, the defined shape range does not match the input placeholder.

    To avoid this problem, when the number of input placeholders is greater than 10, you are advised to specify the placeholder names in the network script. In this case, the placeholders are named based on the specified names to associate the shape ranges with the placeholder names.

  • This option cannot be used together with dynamic_dims. If both are configured, dynamic_dims takes precedence and this option is ignored.

Training/Online inference

graph_memory_max_size

Sizes of the network static memory and the maximum dynamic memory (used in earlier versions).

In the current version, this parameter does not take effect. The system dynamically allocates memory resources based on the actual memory usage of the network.

Training/Online inference

variable_memory_max_size

Size of the variable memory (used in earlier versions).

In the current version, this parameter does not take effect. The system dynamically allocates memory resources based on the actual memory usage of the network.

Training/Online inference