Model Inference Performance Tuning Strategies

During network-wide inference, the model inference performance may not meet the expectation due to problems such as operator adaptation and data read/write on AI processor. This section describes the performance tuning process during model inference. Before tuning inference performance of your model, first debug its inference functionality.

Key tools required for tuning are the Ascend Optimization Engine (AOE) and Ascend Model Compression Toolkit (AMCT). The tuning also involves operations such as model conversion, time consumption recoding, and performance bottleneck analysis, which require tools such as the model conversion tool Ascend Tensor Compiler (ATC), performance data collection tool, and accuracy comparison tool.