Tuning Result Viewing

If the tuning result is as follows, the tuning is complete. For details about the tuned custom repository, see Custom Repository. For details about the tuning result files, see Operator Tuning Result File.

[Aoe]<xxxx> Tuning process finished.     //xxxx indicates the operator name.
[Aoe]Aoe process finished,cost time 1109s.              //Tuning complete.

Custom Repository

After tuning is complete, if the conditions for generating a custom repository are met (see Figure 3), a custom repository is generated. The priority of the paths for storing the custom repository is: TUNE_BANK_PATH > ASCEND_CACHE_PATH > default path. For details about TUNE_BANK_PATH and ASCEND_CACHE_PATH, see the Environment Variables.

If TUNE_BANK_PATH and ASCEND_CACHE_PATH are not configured, the custom repository is stored in ${HOME}/Ascend/latest/data/aoe/custom/op/${soc_version} by default. You can run the env command to check whether they are configured.

For details about how to use the tuned custom repository, see Usage of Tuned Custom Repositories.

Operator Tuning Result File

The priority of the paths for storing the operator tuning result file is: ASCEND_WORK_PATH > default path (tuning working directory). To be specific, if ASCEND_WORK_PATH is not configured, this file is stored in the default path (tuning working directory). You can run the env command to check whether ASCEND_WORK_PATH is configured. For details about ASCEND_WORK_PATH, see the Environment Variables.

During tuning, the result file generated in real time is named aoe_result_opat_${timestamp}_${pidxxx}.json, which records the information about the tuned operators. ${timestamp} is in the format of YYYYMMDD_HHMMSSMS. The variable ${pidxxx} indicates the process ID.

The content format is as follows. Multiple tuning tasks can be included. For details about the fields, see Table 1. tid indicates the thread ID.

{
   "report_${timestamp}_${tid}": [
    {
      "basic": {
        "tuning_name": "Tuning task name",
        "tuning_time(s)": 26
      }
    },
    {
      "OPAT": {
        "opat_tuning_result": "tuning successful",
      "repo_modified_operators": [
        {
          "op_name": "softmax",
          "op_type": "SoftmaxV2",
          "tune_performance": {
            "Format": {
              "performance_after_tune(us)": 99,
              "performance_before_tune(us)": 134,
              "performance_improvement": "35.35%",
              "update_mode": "add"
            }
          }
        },
       .......
        {
          "op_name": "Conv_125",
          "op_type": "Conv2D",
          "tune_performance": {
            "Schedule": {
              "performance_after_tune(us)": 72.046,
              "performance_before_tune(us)": 72.055,
              "performance_improvement": "0.01%",
              "update_mode": "add"
            }
          }
        }
      ],
      "repo_summary": {
        "repo_add_num": 19,
        "repo_hit_num": 0,
        "repo_reserved_num": 0,
        "repo_unsatisfied_num": 2,
        "repo_update_num": 0,
        "total_num": 21
      }
    }
  }
],
  "report_${timestamp}_${tid}": [
   ........
   .....

If the tuning fails (tuning failed is displayed in opat_tuning_result), the op_name list of the operators that fail to be tuned is also displayed.

      "tuning_failed_operators": [
        "res4a_branch1"
       ]

**Table 1** Fields
Field Name				Description
basic
-	tuning_name	-	-	Tuning task name.
-	tuning_time(s)	-	-	Tuning duration, in seconds. This field is not recorded in tuning interruption scenarios (such as coredump and OOM).
OPAT NOTE: If no operator is available to be tuned, information in this segment does not exist.
-	opat_tuning_result	-	-	Tuning result, which can be "tuning successful" during a tuning success, "tuning failed" during a tuning failure, or "tune tuning incomplete" during an incomplete tuning or abnormal exit.
-	repo_modified_operators	-	-	Details about operators whose tiling policies are added or updated after tuning.
-	-	op_name	-	Operator name.
-	-	op_type	-	Operator type. There can be one or more types. If there are multiple types, use [] to enclose them.
-	-	tune_performance	-	Detailed information about operator performance improvement.
-	-	Format, Schedule, or Impl	-	Operator tuning mode. The options are as follows: Format: This field is available only when Format is enabled during operator tuning and the performance is improved through the Format tuning. Schedule: This field is available only when the performance is improved through the Schedule tuning. Impl: This field is available only when the performance is improved through the Impl tuning.
-	-	-	performance_after_tune(us)	Operator execution time after tuning, in μs.
-	-	-	performance_before_tune(us)	Operator execution time before tuning, in μs.
-	-	-	performance_improvement	Percentage of reduced operator execution time after tuning.
-	-	-	update_mode	Update mode of the operator tiling policies. The options are as follows: add: adds operator tiling policies. update: updates operator tiling policies.
NOTE: The information from op_name to update_mode is displayed for each operator whose tiling policies are added or updated.
-	repo_summary	-	-	Information about operators in each state during tuning.
-	-	repo_add_num	-	Number of the titling policies that are not in the repository before tuning and are added to the repository after tuning.
-	-	repo_hit_num	-	Number of the titling policies that are in the repository during tuning.
-	-	repo_reserved_num	-	Number of the titling policies that are in the repository before tuning and remain unchanged after tuning.
-	-	repo_unsatisfied_num	-	Number of the titling policies that are not in the repository before tuning and are not written into the repository after tuning.
-	-	repo_update_num	-	Number of the titling policies that are in the repository before tuning and are updated after tuning.
-	-	total_num	-	Total number of titling policies that are tuned in the tuning task. repo_hit_num=repo_update_num+repo_reserved_num total_num=repo_add_num+repo_hit_num+repo_unsatisfied_num
-	tuning_failed_operators	-	-	op_name list of operators that fail to be tuned. NOTE: This field is optional. It is recorded only when the value of opat_tuning_result is tuning failed.

Parent topic: Offline Tuning in PyTorch-based Training Scenarios