--op_debug_level

Applicability

Product

Supported (√/x)

Atlas A3 training product/Atlas A3 inference product

Atlas A2 training product/Atlas A2 inference product

Atlas 200I/500 A2 inference product

Atlas inference product

Atlas training product

Description

Enables TBE operator debug during operator compilation.

See Also

None

Arguments

Arguments:

  • 0: disables operator debug. The operator compilation folder kernel_meta is not generated under the aoe command execution directory.
  • 1: enables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-cce mapping file *_loc.json) are generated in the folder. These files are used to locate AI Core errors with the AI Core Error Analyzer.
  • 2: enables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-cce mapping file *_loc.json) are generated in the folder. These files are used to locate AI Core errors with the AI Core Error Analyzer. Setting this parameter to 2 also disables the compilation optimization switch and enables the CCEC debug function (the CCEC compiler option is set to -O0-g).
  • 3: disables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file) and .json file (operator description file) are generated in the folder. You can refer to these files when analyzing operator errors.
  • 4: disables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the *.o file (operator binary file), *.json file (operator description file), TBE instruction mapping file (operator file *.cce), and UB fusion compute description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and accuracy comparison during operator error analysis.

To generate the kernel_meta folder, you must have the read and write permissions on the directory where the aoe command is executed.

Default: 0

Restrictions: To perform model tuning, set this option to 0, 3, or 4. To locate AI Core errors, set this option to 1 or 2. However, the network performance will deteriorate as the debugging function is configured.

Suggestions and Benefits

None

Examples

--op_debug_level=1

Dependencies and Restrictions

None