Tuning in TensorFlow-based Online Inference Scenarios

Both subgraph tuning and operator tuning are supported in TensorFlow-based online inference scenarios. The online tuning process here is basically the same as that in TensorFlow-based training scenarios. For details, see Online Tuning in TensorFlow-based Training Scenarios.

The differences between tuning in TensorFlow-based online inference scenarios and online tuning in TensorFlow-based training scenarios are as follows:

• During tuning by setting environment variables, the online inference script is executed for online inference, and the training script is executed for TensorFlow-based training.
• During tuning by modifying the script, in online inference scenarios, only the sess.run mode is supported, and session configuration options aoe_mode and work_path are used to enable AOE tuning. Only TensorFlow 1.15 and TensorFlow 2.6.5 are supported.