Documentation

--op_debug_level

Description

Enables operator debug.

See Also

If you want to specify the path for storing the process file of operator build, use --op_debug_level (non-zero value) together with --debug_dir.

Argument

Argument:

  • 0 (default): Disables operator debug. The operator build folder kernel_meta is not generated in the current execution path.
  • 1: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for later analysis of AI Core errors.
  • 2: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for later analysis of AI Core errors. Setting this option to 2 also disables build optimization and enables the CCE compiler debug function (the CCE compiler option is set to -O0-g).
  • 3: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the .o file (operator binary file) and .json file (operator description file) are generated in the folder. You can refer to these files when analyzing operator errors.
  • 4: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the .o file (operator binary file), .json file (operator description file), TBE instruction mapping file (operator file *.cce), and UB fusion description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and precision comparison during operator error analysis.

Restrictions:

  • To perform model conversion, set this option to 0, 3, or 4. To locate AI Core errors, set this option to 1 or 2. If this option is set to 1 or 2, the network performance deteriorates because of the extra debug logic.
  • If --op_debug_level is set to 0 and --op_debug_config is set, the operator build directory kernel_meta is still retained in the directory where the atc command is executed.
  • If --op_debug_level is set to 0 and the NPU_COLLECT_PATH environment variable is set, the build directory kernel_meta is always retained. If the ASCEND_WORK_PATH environment variable is set, the build directory is retained in the path specified by the environment variable. If the ASCEND_WORK_PATH environment variable does not exist, the build directory is retained in the current execution path.
  • If --op_debug_level is set to 2 (that is, CCEC compilation is enabled), the size of the operator kernel file (*.o file) increases. In the dynamic shape scenario, all possible shape scenarios are traversed during operator build, which may cause operator build failures due to large operator kernel files. In this case, do not enable the CCE compiler options.

    If a build failure is caused by the large operator kernel file, the following log is displayed:

    message:link error ld.lld: error: InputSection too large for range extension thunk ./kernel_meta_xxxxx.o
  • When the debug function is enabled, if the model contains the following MC2 operators, the *.o, *.json, and *.cce files of the operators are not generated in the kernel_meta directory.

    MatMulAllReduce

    MatMulAllReduceAddRmsNorm

    AllGatherMatMul

    MatMulReduceScatter

    AlltoAllAllGatherBatchMatMul

    BatchMatMulReduceScatterAlltoAll

Suggestions and Benefits

None

Example

--op_debug_level=1

Applicability

Atlas 200/300/500 Inference Product

Atlas Training Series Product

Restrictions

  • In addition to --debug_dir, the environment variable ASCEND_WORK_PATH can be used to set the path for storing the debug file generated by operator build. The priorities of the configuration methods are as follows: --debug_dir > ASCEND_WORK_PATH > default storage path.
  • This option takes precedence over tbe_debug_level passed to the build API of TBE DSL or BuildCCE API of TBE TIK.
Parent topic: Debug Options