Subgraph/Operator Tuning

Tuning Precautions

1. Ensure that the training script can be successfully executed on the Ascend AI Processor and the function and accuracy meet the expectation.
2. You are not advised binding the training process to a specified CPU. Use the default CPU scheduling policy. Otherwise, the tuning effect may be affected.
3. To improve the tuning efficiency, you are advised to control the number of training steps as much as possible. Generally, a complete graph execution process can be completed through one step. Ensure that all operators in the graph can be traversed for tuning.
4. Currently, only static operators are supported. Dynamic operators are not supported.
5. AOE does not allow different users to use the same device for tuning at the same time.
6. Before tuning, disable the profiling function and the static memory allocation mode (that is, set static_memory_policy to 0, for example, custom_op.parameter_map["static_memory_policy"].i = 0) to avoid affecting the tuning result. For details about how to disable the profiling function, see the Performance Tuning Tool User Guide .
7. Tuning is not supported in multi-device scenarios.
8. To perform tuning in the single-device scenario, ensure that the following conditions are met.
   • Available disk space in the home directory of the user who performs tuning: ≥ 20 GB
   • Available memory ≥ Memory required for model training x TUNING_PARALLEL_NUM. For details about TUNING_PARALLEL_NUM, see Configuration File.
   • Recommended quantity of Host CPUs during operator tuning: ≥ Number of processes in the training script x (TE_PARALLEL_COMPILER + TUNING_PARALLEL_NUM + min(Number of CPU cores/2, 8) + 50). For details about TE_PARALLEL_COMPILER and TUNING_PARALLEL_NUM, see Table 1 and Configuration File.
   • Recommended quantity of Host CPUs during subgraph tuning: ≥ Number of processes in the training script x (2 x TUNING_PARALLEL_NUM + TE_PARALLEL_COMPILER). For details about TE_PARALLEL_COMPILER and TUNING_PARALLEL_NUM, see Table 1 and Configuration File.
   • Number of device cores ≥ Maximum number of cores used by all operators in the model
   • Device memory: related to the model and model memory overcommitment.

Environment Variable Configuration

• You can write the commands for configuring environment variables to the custom script for future use.

• In addition to using environment variables to specify the tuning mode, you can also modify the corresponding parameters in the training script to specify the tuning mode. For details, see Configuring the Tuning Mode by Modifying the Training Script (TensorFlow 1.15). Considering the operation complexity, you are advised to specify the tuning mode by setting the environment variable.

Tuning Procedure

Run the training script to automatically tune subgraphs or operators based on the specified tuning mode.

• To specify the tuning path, you can modify work_path or aoe_config.work_path in the training script before training.
• To tune a specified operator, you can modify aoe_config_file or aoe_config.aoe_config_file in the training script before training.

Tuning Result Viewing

1
2
3
4

# Enable TFAdapter tuning.
in tune mode, training graph handled by tools.
# Start the tool for tuning.
Aoe tuning graph.

After the tuning is complete, if the conditions for generating a custom repository are met (see Figure 2 and Figure 3), a custom repository is generated.

Performance Verification

After subgraph tuning is complete, restore the code and use the tuned custom repository to perform training again to check whether the performance is improved. For details about how to use the custom repository, see Usage of Tuned Custom Repositories.

After operator tuning is complete, restore the code and refresh the operator compilation cache. That is, set op_compiler_cache_mode to force (see the TensorFlow 1.15 Model Porting Guide) and perform training again using the tuned custom repository to check whether the performance is improved. For details about how to use the custom repository, see Usage of Tuned Custom Repositories.

Operator Tuning Result File

The priority of the paths for storing the operator tuning result file is: ASCEND_WORK_PATH > default path (tuning working directory). To be specific, if ASCEND_WORK_PATH is not configured, this file is stored in the default path (tuning working directory). You can run the env command to check whether ASCEND_WORK_PATH is configured. For details about ASCEND_WORK_PATH, see the Environment Variables.

During tuning, the result file generated in real time is named aoe_result_opat_${timestamp}_${pidxxx}.json, which records the information about the tuned operators. ${timestamp} is in the format of YYYYMMDD_HHMMSSMS. The variable ${pidxxx} indicates the process ID.

The content format is as follows. Multiple tuning tasks can be included. For details about the fields, see Table 2. tid indicates the thread ID.

{
  "report_${timestamp}_${tid}": [
    {
      "basic": {
        "tuning_name": "Tuning task name",
        "tuning_time(s)": 44
      }
    },
    {
      "OPAT": {
        "opat_tuning_result": "tuning successful",
        "repo_modified_operators": [
          {
            "op_name": "bert/encoder/layer_10/attention/self/Softmax_OPAT_0",
            "op_type": "SoftmaxV2",
            "tune_performance": {
              "Format": {
                "performance_after_tune(us)": 26.876,
                "performance_before_tune(us)": 58.781,
                "performance_improvement": "118.71%",
                "update_mode": "add"
              }
            }
          },
          {
            "op_name": "bert/encoder/layer_8/attention/output/dense/MatMulbert/encoder/layer_8/attention/output/add",
            "op_type": "MatMulV2",
            "tune_performance": {
              "Schedule": {
                "performance_after_tune(us)": 15.71,
                "performance_before_tune(us)": 16.71,
                "performance_improvement": "6.37%",
                "update_mode": "add"
              }
            }
          }
        ],
        "repo_summary": {
          "repo_add_num": 2,
          "repo_hit_num": 10,
          "repo_reserved_num": 10,
          "repo_unsatisfied_num": 1,
          "repo_update_num": 0,
          "total_num": 13
        }
      }
    }
  ],
  "report_${timestamp}_${tid}": [
   ........
   .....

If the tuning fails (tuning failed is displayed in opat_tuning_result), the op_name list of the operators that fail to be tuned is also displayed.

      "tuning_failed_operators": [
        "res4a_branch1"
       ]

Subgraph Tuning Result File

The priority of the paths for storing the subgraph tuning result file is: ASCEND_WORK_PATH > default path (tuning working directory). To be specific, if ASCEND_WORK_PATH is not configured, this file is stored in the default path (tuning working directory). You can run the env command to check whether ASCEND_WORK_PATH is configured. For details about ASCEND_WORK_PATH, see the Environment Variables.

During tuning, the result file generated in real time is named aoe_result_sgat_${timestamp}_${pidxxx}.json, which records the information about the tuned subgraphs. ${timestamp} is in the format of YYYYMMDD_HHMMSSMS. The variable ${pidxxx} indicates the process ID.

The content format is as follows. Multiple tuning tasks can be included. For details about the fields, see Table 3. tid indicates the thread ID.

"report_${timestamp}_${tid}": [
    {
      "basic": {
        "tuning_name": "Tuning task name",
        "tuning_time(s)": 19
      }
    },
    {
      "SGAT": {
        "model_baseline_performance(ms)": 5.600486,
        "model_performance_improvement": "55.11%",
        "model_result_performance(ms)": 3.610442,
        "repo_modified_subgraphs": {
          "add_repo_subgraphs": [
            {
              "performance_after_tune(ms)": 3.573203,
              "performance_before_tune(ms)": 5.58434,
              "performance_improvement": "56.28%",
              "repo_key": "1024942313106047484"
            }
          ]
          "update_repo_subgraphs": [
            {
              "performance_after_tune(ms)": 2.573203,
              "performance_before_tune(ms)": 4.58434,
              "performance_improvement": "78.15%",
              "repo_key": "1024942313106057586"
            }
          ]
        },
        "repo_summary": {
          "repo_add_num": 1,
          "repo_hit_num": 1,
          "repo_reserved_num": 0,
          "repo_unsatisfied_num": 120,
          "repo_update_num": 1,
          "total_num": 121
        }
      }
    }
  ],
  "report_${timestamp}_${tid}": [
   .......
   .......