Performance Tuning

ge.exec.op_precision_mode

Sets the precision mode of one or more specified operators during internal processing. This parameter is used to transfer the customized precision mode configuration file op_precision.ini to set different precision modes for different operators.

Set the precision mode based on the operator type (low priority) or node name (high priority) in each row in the .ini file.

The following precision modes can be set in the configuration file:

• high_precision
• high_performance
• enable_float_32_execution: The FP32 data type is used for internal processing of operators. In this scenario, the FP32 data type is not automatically converted to the HF32 data type. If you are using the HF32 data type for computation and find that the accuracy drop exceeds your expectation, you can enable this configuration to specify the use of FP32 for internal computation of certain operators in order to maintain accuracy.

  This option is supported only by the following products:

  Ascend 950PR/Ascend 950DT

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

• enable_hi_float_32_execution: The HF32 data type is used for internal processing of operators. After this option is enabled, the FP32 data type is automatically converted to HF32. This can reduce the space occupied by data and improve performance.

  This option is not supported in the current version.

• support_out_of_bound_index: indicates that the out-of-bounds verification is performed on the indices of the gather, scatter, and segment operators. The verification deteriorates the operator execution performance.
• keep_fp16: The FP16 data type is used for internal processing of operators. In this mode, the FP16 data type is not automatically converted to FP32. If FP32-based computation fails to meet performance expectations and high accuracy is not required, you can enable the keep_fp16 mode. This low-precision mode trades accuracy for performance and is not recommended.
• super_performance: Indicates ultra-high performance. Compared with high performance, the algorithm calculation formula is optimized.

You can view the precision or performance mode supported by an operator in the opp/built-in/op_impl/ai_core/tbe/impl_mode/all_ops_impl_mode.ini file in the file storage path with the CANN software installed.

A configuration example for the op_precision.ini file is as follows:

[ByOpType]
optype1=high_precision
optype2=high_performance
optype3=enable_hi_float_32_execution
optype4=support_out_of_bound_index

[ByNodeName]
nodename1=high_precision
nodename2=high_performance
nodename3=enable_hi_float_32_execution
nodename4=support_out_of_bound_index

Configuration example:

{"ge.exec.op_precision_mode", "$HOME/conf/op_precision.ini"};

Mandatory/Optional: optional

Effective level: global

ge.exec.variable_acc

Sets whether to enable variable format optimization.

Parameter values:

• True (default): enabled
• False: disabled

To improve training efficiency, the format of the variables is converted to a format more compatible with the AI processor during variable initialization performed by the network. Enable or disable this function as needed.

Restrictions:

When this function is enabled, ge.AllowMultiGraphParallelCompile cannot be set to 1. Otherwise, an error is reported during verification.

Configuration example:

{"ge.exec.variable_acc", "True"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.graphMaxParallelModelNum

Controls the maximum number of models that can be concurrently loaded. In graph execution mode, a graph can be concurrently loaded and executed by multiple models on the same device.

Parameter values:

1 to INT32_MAX. The default value is 8.

Configuration example:

{"ge.graphMaxParallelModelNum", "8"};

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.tiling_schedule_optimize

Sets whether to enable the optimization for tiling offload scheduling.

As internal storage of the AI Cores in the NPU cannot store all the input and output data of operators, the input data is tiled into different parts. The first part is transferred in, computed, and then transferred out, so does the next part. This process is called tiling. Then, a computation program, called tiling implementation, determines tiling parameters (such as the block size transferred each time and the total number of cycles) based on operator information such as shape. The AI Core is not good at scalar computation in the tiling implementation. Tiling implementation is generally executed on the CPU on the host. However, tiling implementation is executed on the device when the following conditions are met:

1. The model is static-shape.
2. Operators in the model, such as the FusedInferAttentionScore and IncreFlashAttention fused operators, support tiling offload.
3. Operators that support tiling offload have dependency requirements: tiling computation must rely on runtime results produced by the previous operator executing on the device. If the dependent value comes from a Const node, tiling offload is not required, and tiling is completed at build time.

This parameter is supported only by the following products:

Atlas A2 training products/Atlas A2 inference products

Atlas A3 training products/Atlas A3 inference products

Atlas inference products

Parameter values:

• 0 (default): Tiling offload is disabled.
• 1: Tiling offload is enabled.

Configuration example:

{"ge.tiling_schedule_optimize", "0"};

Mandatory/Optional: optional

Effective level: global, session, and graph