Performance Tuning

OP_PRECISION_MODE

Sets the precision mode of one or more specified operators during internal processing. This parameter is used to transfer the customized precision mode configuration file op_precision.ini to set different precision modes for different operators. The corresponding options parameter is ge.exec.op_precision_mode.

Configuration example:

{ge::ir_option::OP_PRECISION_MODE, "op_precision.ini"}

The following precision modes can be set in the configuration file:

• high_precision
• high_performance
• enable_float_32_execution: The FP32 data type is used for internal processing of operators. In this scenario, the FP32 data type is not automatically converted to the HF32 data type. If you are using the HF32 data type for computation and find that the accuracy drop exceeds your expectation, you can enable this configuration to specify the use of FP32 for internal computation of certain operators in order to maintain accuracy.

  This option is supported only by the following products:

  Ascend 950PR/Ascend 950DT

  Atlas A2 training products/Atlas A2 inference products

  Atlas A3 training products/Atlas A3 inference products

• enable_hi_float_32_execution: The HF32 data type is used for internal processing of operators. After this option is enabled, the FP32 data type is automatically converted to HF32. This can reduce the space occupied by data and improve performance.

  This option is not supported in the current version.

• support_out_of_bound_index: indicates that the out-of-bounds verification is performed on the indices of the gather, scatter, and segment operators. The verification deteriorates the operator execution performance.
• keep_fp16: The FP16 data type is used for internal processing of operators. In this mode, the FP16 data type is not automatically converted to FP32. If FP32-based computation fails to meet performance expectations and high accuracy is not required, you can enable the keep_fp16 mode. This low-precision mode trades accuracy for performance and is not recommended.
• super_performance: Indicates ultra-high performance. Compared with high performance, the algorithm calculation formula is optimized.

You can view the precision or performance mode supported by an operator in the opp/built-in/op_impl/ai_core/tbe/impl_mode/all_ops_impl_mode.ini file in the file storage path with the CANN software installed.

Sample: Set the precision mode based on the operator type (low priority) or node name (high priority) in each row in the .ini file.

[ByOpType]
optype1=high_precision
optype2=high_performance
optype3=enable_hi_float_32_execution
optype4=support_out_of_bound_index

[ByNodeName]
nodename1=high_precision
nodename2=high_performance
nodename3=enable_hi_float_32_execution
nodename4=support_out_of_bound_index

Restrictions:

• This parameter is mutually exclusive with OP_SELECT_IMPL_MODE and OPTYPELIST_FOR_IMPLMODE. If they are all specified, OP_PRECISION_MODE takes precedence.
• For the same operator, if enable_hi_float_32_execution or enable_float_32_execution is configured using OP_PRECISION_MODE, you are advised not to use this parameter together with ALLOW_HF32. If they are used together, the priority is as follows:

  op_precision_mode(ByNodeName) > allow_hf32 > op_precision_mode(ByOpType)

• You are advised not to set this parameter. It is used if you need to adjust the precision of a specific operator using the configuration .ini file in the case that you fail to obtain optimal network performance or accuracy in the high-performance or high-precision mode.

Applicable products:

All processors

TILING_SCHEDULE_OPTIMIZE

Sets whether to enable the optimization for tiling offload scheduling. The corresponding options parameter is ge.tiling_schedule_optimize.

As internal storage of the AI Cores in the NPU cannot store all the input and output data of operators, the input data is tiled into different parts. The first part is transferred in, computed, and then transferred out, so does the next part. This process is called tiling. Then, a computation program, called tiling implementation, determines tiling parameters (such as the block size transferred each time and the total number of cycles) based on operator information such as shape. The AI Core is not good at scalar computation in the tiling implementation. Tiling implementation is generally executed on the CPU on the host. However, tiling implementation is executed on the device when the following conditions are met:

1. The model is static-shape.
2. Operators in the model, such as the FusedInferAttentionScore and IncreFlashAttention fused operators, support tiling offload.
3. Operators that support tiling offload have dependency requirements: tiling computation must rely on runtime results produced by the previous operator executing on the device. If the dependent value comes from a Const node, tiling offload is not required, and tiling is completed at build time.

Parameter values:

• 0 (default): Tiling offload is disabled.
• 1: Tiling offload is enabled.

Configuration example:

{ge::ir_option::TILING_SCHEDULE_OPTIMIZE, "1"}

Applicable products:

• Ascend 950PR/Ascend 950DT: supported
• Atlas A3 training products/Atlas A3 inference products: supported
• Atlas A2 training products/Atlas A2 inference products: supported
• Atlas 200I/500 A2 inference products: not supported
• Atlas inference products: supported
• Atlas training products: not supported