Performance Tuning

ENABLE_SMALL_CHANNEL

Sets whether to enable small channel tuning. If small channel tuning is enabled, performance benefits are generated at the convolutional layers with channel size ≤ 4. You are advised to enable this function in inference scenarios. The corresponding options parameter is ge.enableSmallChannel.

Parameter values:

• 0 (default): disabled
• 1: enabled

Configuration example:

{ge::ir_option::ENABLE_SMALL_CHANNEL, "1"}

Applicable products:

All processors

TILING_SCHEDULE_OPTIMIZE

Sets whether to enable the optimization for tiling offload scheduling. The corresponding options parameter is ge.tiling_schedule_optimize.

As internal storage of the AI Cores in the NPU cannot store all the input and output data of operators, the input data is tiled into different parts. The first part is transferred in, computed, and then transferred out, so does the next part. This process is called tiling. Then, a computation program, called tiling implementation, determines tiling parameters (such as the block size transferred each time and the total number of cycles) based on operator information such as shape. The AI Core is not good at scalar computation in the tiling implementation. Tiling implementation is generally executed on the CPU on the host. However, tiling implementation is executed on the device when the following conditions are met:

1. The model is static-shape.
2. Operators in the model, such as the FusedInferAttentionScore and IncreFlashAttention fused operators, support tiling offload.
3. Operators that support tiling offload have dependency requirements: tiling computation must rely on runtime results produced by the previous operator executing on the device. If the dependent value comes from a Const node, tiling offload is not required, and tiling is completed at build time.

Parameter values:

• 0 (default): Tiling offload is disabled.
• 1: Tiling offload is enabled.

Configuration example:

{ge::ir_option::TILING_SCHEDULE_OPTIMIZE, "1"}

Applicable products:

• Ascend 950PR/Ascend 950DT: supported
• Atlas A3 training products/Atlas A3 inference products: supported
• Atlas A2 training products/Atlas A2 inference products: supported
• Atlas 200I/500 A2 inference products: not supported
• Atlas inference products: supported
• Atlas training products: not supported