--op_debug_level

Applicable Products

All processors

Description

Enables operator debug.

See Also

If you need to specify the path for storing the operator build intermediate files, use --op_debug_level (set to a non-zero value) together with --debug_dir.

Arguments

Argument:

• 0 (default): Disables operator debug. The operator build folder kernel_meta is not generated in the current execution path.
• 1: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors.

  For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

• 2: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors. Setting this option to 2 also disables build optimization and enables the CCE compiler debug function (the CCE compiler option is set to -O0-g).

  For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

• 3: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o) and operator description file (.json) are generated in the folder. You can refer to these files when analyzing operator errors.
• 4: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), TBE instruction mapping file (operator CCE file *.cce), and UB fusion description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and accuracy comparison during operator error analysis.

  For the Ascend 950PR / Ascend 950DT , neither TBE instruction mapping files nor UB fusion compute description files are generated.

Restrictions:

• To perform model conversion, set this option to 0, 3, or 4. To locate AI Core errors, set this option to 1 or 2. If this option is set to 1 or 2, the network performance deteriorates because of the extra debug logic.
• If --op_debug_level is set to 0 and --op_debug_config is set, the operator build directory kernel_meta is still retained in the directory where the atc command is executed.
• If --op_debug_level is set to 0 and the NPU_COLLECT_PATH environment variable is set, the build directory kernel_meta is always retained. If the ASCEND_WORK_PATH environment variable is set, the build directory is retained in the path specified by the environment variable. If the ASCEND_WORK_PATH environment variable does not exist, the build directory is retained in the current execution path.
• When this option is set to 2, the CCE compiler is enabled, and the size of the operator kernel file (*.o) increases. In dynamic-shape scenarios, all possible shape cases are traversed during operator build, which may result in operator build failures caused by oversized operator kernel files. For such scenarios, you are advised not to enable the CCE compiler options.

  The following is an example error log generated when operator build fails due to oversized operator kernel files:

  message:link error ld.lld: error: InputSection too large for range extension thunk ./kernel_meta_xxxxx.o

• When the debug function is enabled, if the model contains the following MC2 operators, the *.o, *.json, and *.cce files of these operators are not generated in the operator build folder kernel_meta.

  MatMulAllReduce

  MatMulAllReduceAddRmsNorm

  AllGatherMatMul

  MatMulReduceScatter

  AlltoAllAllGatherBatchMatMul

  BatchMatMulReduceScatterAlltoAll

Suggestions and Benefits

None

Example

--op_debug_level=1

Restrictions

• In addition to --debug_dir, the environment variable ASCEND_WORK_PATH can be used to set the path for storing the debug file generated by operator build. The priorities of the configuration methods are as follows: --debug_dir > ASCEND_WORK_PATH > default storage path.
• This option takes precedence over tbe_debug_level passed to the build API of TBE DSL or BuildCCE API of TBE TIK.