--op_debug_level

Applicability

Product

Supported

Atlas A3 training products/Atlas A3 inference products

Atlas A2 training products/Atlas A2 inference products

Atlas 200I/500 A2 inference products

Atlas inference products

Atlas training products

Description

Enables TBE operator debug during operator compilation.

See Also

None

Arguments

Arguments:

  • 0: disables operator debug. The operator compilation folder kernel_meta is not generated under the aoe command execution directory.
  • 1: enables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-cce mapping file *_loc.json) are generated in the folder. These files are used to locate AI Core errors with the AI Core Error Analyzer.
  • 2: enables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file), .json file (operator description file), and TBE instruction mapping files (operator file *.cce and python-cce mapping file *_loc.json) are generated in the folder. These files are used to locate AI Core errors with the AI Core Error Analyzer. Setting this parameter to 2 also disables the compilation optimization switch and enables the CCEC debug function (the CCEC compiler option is set to -O0-g).
  • 3: disables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file) and .json file (operator description file) are generated in the folder. You can refer to these files when analyzing operator errors.
  • 4: disables operator debug. The kernel_meta folder is generated in the directory where the aoe command is executed, and the .o file (operator binary file), .json file (operator description file), TBE instruction mapping file (operator file *.cce), and UB fusion compute description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and accuracy comparison during operator error analysis.

To generate the kernel_meta folder, you must have the read and write permissions on the directory where the aoe command is executed.

Default: 0

Restrictions: To perform model tuning, set this option to 0, 3, or 4. To locate AI Core errors, set this option to 1 or 2. However, the network performance will deteriorate as the debugging function is configured.

Suggestions and Benefits

None

Examples

--op_debug_level=1

Dependencies and Restrictions

None