Operator Tuning

Dump operator information.
Before operator tuning, obtain the operator information file (.json file) in the model, which contains the shape, dtype, and format information of the operator. Currently, there are two methods to dump the operator .json file. Choose one as required.
- When using Python APIs of PyTorch for programming, you can call the Ascend PyTorch Profiler API to dump the operator .json file. For details, see "Ascend PyTorch Profiler" in Profiling Instructions.
  1. Before training, enable the operator information statistics function in the extended option experimental_config by setting record_op_args to True.
  2. View the result file of profile data collected during PyTorch-based training.
    After the training is complete, the dumped operator .json file is stored in the {worker_name}_{timestamp}_ascend_pt_op_args/{pid} directory by default.
- When using acl APIs for programming, you can dump the operator information file to a specified directory by calling aclopStartDumpArgs and aclopStopDumpArgs.
Compile the static kernel package.
In any directory, run the corresponding command as the running user (for example, HwHiAiUser) to compile the operator:
- Command example of static compilation (default mode):
```
op_compiler --op_params_dir=<dump_dir>  --soc_version=<soc_version> --log=info --job=8 --output=<output_dir>
```
- Command example of compilation (tuning mode):
```
op_compiler --op_params_dir=<dump_dir>  --soc_version=<soc_version> --log=info --job=8 --compile_mode=tune --output=<output_dir>
```
The key parameters are described as follows. You can set them as required by referring to "Command-Line Options" in Operator Compilation Tool User Guide.
- --op_params_dir: -p for short. This option is required, and specifies the absolute or relative path where the operator information file exported by the dump tool is stored.
- --soc_version: -v for short. This option is required when the operator compilation function is performed, and specifies the model of the Ascend AI Processor during operator compilation.
  
  If the soc_version of the current device cannot be determined, run the npu-smi info command on the server where the NPU driver package is installed and add Ascend before the obtained value of Name. For example, if the value of Name is xxxyy, the actual soc_version is Ascendxxxyy.
- --log: -l for short. This option is optional and specifies the log level during operator compilation. It can be set to debug, info, warning, error, or null (default).
- --job: -j for short. This option is optional and specifies the number of working processes during compilation. The minimum value is 1 and the default value is 16.
- --compile_mode: -m for short. This option is optional and can be set to tune, indicating that the tuning mode is enabled and the tuning and compilation process is executed. When this option is not used, the default compilation process is executed.
- --output: -o for short. This option is optional, and specifies the absolute or relative path and name of the output installation package, for example, xxx/xxx/xxx.run. If no path is specified, the installation package is generated in the current path. If no package name is specified, the default name static_kernel_${datetime}_${pid}.run is used.
If you see information similar to the following, the compilation is successful:
```
generate run package static_kernel_${datetime}_${pid}.run success
```
- You can use --count and -p together to count the number of operator information .json files in the directory specified by -p.
  An example is as follows.
```
op_compiler -p <dump_dir> --count
```
  Operation information can be dumped only for dynamic shapes. After the static kernel package is installed, the information of the corresponding operator will not be dumped. If the network is adjusted, you can check whether the static kernel package matches the current network by comparing the number of dumped .json files before and after the adjustment.
  
  That is, perform the dump operation before and after the network adjustment and then use the --count option to count the number of .json files generated after the dump operation. If the count after the adjustment is greater than that before the adjustment, some operators in the static kernel package no longer match the current network. In this case, you can choose either of the following methods:
  - Uninstall the static kernel package, perform the dump operation again, and compile and install the new static kernel package.
  - Use the current static kernel package still. In this case, a dynamic process is started for unmatched operators, and performance improvement cannot be achieved.
- Compilation commands cannot be executed in dump_dir.

Install the static kernel package.

Go to the directory where the static_kernel_${datetime}_${pid}.run package is stored and run the package as the running user (for example, HwHiAiUser).

./static_kernel_${datetime}_${pid}.run

If you see information similar to the following, the installation is successful.

Verifying archive integrity...  100%   SHA256 checksums are OK. All good.
Uncompressing STATIC KERNEL RUN PACKAGE  100%

Currently, the installation directory cannot be specified. By default, the .run package is installed in ${install_path}/opp/static_kernel. ${install_path} indicates the CANN component directory. Replace it with the actual path.

The default installation path ${install_path}/opp/static_kernel of the .run package uses the default permission 770 (accessed by the current user and users in the same group). If the static kernel package fails to be installed due to lack of permission, contact the installation user of the CANN package to modify the permission on the static_kernel directory.

The following is an example of the directory structure after the .run package is installed:

|-- ${install_path}/opp/static_kernel
    |-- ai_core
        |-- config                                
           |-- ascendxxxx
               |-- binary_info_config.json            # Indexes of all static kernel packages.
        |-- config.ini                               # Configuration file that records the installation sequence.
        |-- static_kernel_250924103634186357_988879            # Static kernel file with the timestamp 250924103634186357 and process ID 988879.
           |-- ascendxxxx
           |   |-- Add                                 # Directory for storing operator binary files.
           |      |-- static_kernel_Add_float16_NCL_xxxx_d0.json
           |      |-- static_kernel_Add_float16_NCL_xxxx_d1.json 
           |      |-- static_kernel_Add_float16_NCL_xxxx_d0.o
           |      |-- static_kernel_Add_float16_NCL_xxxx_d1.o
           |   |-- xxxx
           |      |-- static_kernel_xxx.json
           |      |-- static_kernel_xxx.o
           |   |-- ......
           |-- config                                # Index of a single static kernel package.
           |   |-- ascendxxxx
           |       |-- binary_info_config.json     
           |-- scripts                               # Common scripts.
           |   |-- ......                         
           |-- uninstall.sh                        # Script for uninstalling a single package.
        |-- static_kernel_xxxx                    # Static kernel files with different timestamps.
        |-- uninstall.sh                           # Script for uninstalling all packages.
        |-- version.info                             # Version information.

Multiple kernel packages can be installed. If two packages contain the same operator kernel, the kernel package installed later is used.

(Optional) Uninstall a single kernel package or all kernel packages.
- Uninstalling a single package
  Go to the directory where the static_kernel_${datetime}_${pid}.run package is installed and run uninstall.sh as the running user (for example, HwHiAiUser).
```
cd ${install_path}/opp/static_kernel/ai_core/static_kernel_${datetime}_${pid}
./uninstall.sh
```
  If the uninstallation is successful, the static_kernel_${datetime}_${pid} folder in the ai_core directory will be deleted.
- Uninstalling all packages
  Go to ${install_path}/opp/static_kernel/ai_core and run uninstall.sh as the running user (for example, HwHiAiUser).
```
cd ${install_path}/opp/static_kernel/ai_core/
./uninstall.sh
```
  If the uninstallation is successful, all content in the ai_core directory will be deleted, and all installed kernel packages will be uninstalled.

Parent topic: Using the Static Kernel to Improve the Model Execution Performance