Performance Optimization

The factors affecting the execution performance of an operator in dynamic-shape scenarios and the workarounds are summarized as follows:

• For dynamic-shape operators, the execution performance is compromised due to frequent condition-based determination. The workaround is to break the supported shapes down into many ranges and generate a separate binary file for each shape range. In this way, the operator implementation is determined based on the actual shape in the execution phase.
• During the calculation of the tiling parameters, the involved scalar computations are typically performed on the CPU with the results moved back to the operator running on the AI Core.
• The operator performance varies with the selected tiling policy. The tiling policy also determines the operator implementation.