Function Debugging

LOG_LEVEL

Sets the log level. The corresponding options parameter is log.

Parameter values:

• debug: Outputs debug, info, warning, and error logs.
• info: Outputs info, warning, and error logs.
• warning: Outputs warning and error logs.
• error: Outputs error logs.
• null (default): No debug log is generated.

Configuration example:

{ge::ir_option::LOG_LEVEL, "debug"}

Applicable products:

All processors

DEBUG_DIR

Directory of the debug-related process files generated during operator build, including the .o (operator binary file), .json (operator description file), and .cce files. The corresponding options parameter is ge.debugDir.

By default, the files are generated in the current directory.

Restrictions:

• If you want to specify the path for storing the process file of operator build, use DEBUG_DIR and OP_DEBUG_LEVEL together. If OP_DEBUG_LEVEL is set to 0, DEBUG_DIR cannot be used.
• In addition to DEBUG_DIR, the ASCEND_WORK_PATH environment variable can be used to set the path for storing the debugging file generated during operator build. The configuration priorities are as follows: DEBUG_DIR > ASCEND_WORK_PATH > the default storage path.

Configuration example:

{ge::ir_option::OP_DEBUG_LEVEL, "1"},
{ge::ir_option::DEBUG_DIR, "/home/test/module/out_debug_info"}

Applicable products:

All processors

OP_DEBUG_LEVEL

Operator debug at operator build time. The corresponding options parameter is ge.opDebugLevel.

If you want to specify the path for storing the process file of operator build, use DEBUG_DIR. If OP_DEBUG_LEVEL is set to 0, DEBUG_DIR does not take effect.

Arguments:

• 0 (default): Disables operator debug. The operator build folder kernel_meta is not generated in the current execution path.
• 1: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors.

  For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

• 2: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors. Setting this option to 2 also disables build optimization and enables the CCE compiler debug function (the CCE compiler option is set to -O0-g).

  For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

• 3: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o) and operator description file (.json) are generated in the folder. You can refer to these files when analyzing operator errors.
• 4: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), TBE instruction mapping file (operator CCE file *.cce), and UB fusion description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and accuracy comparison during operator error analysis.

  For the Ascend 950PR / Ascend 950DT , neither TBE instruction mapping files nor UB fusion compute description files are generated.

Configuration example:

{ge::ir_option::OP_DEBUG_LEVEL, "1"}

Applicable products:

All processors

OP_DEBUG_CONFIG

Global memory check switch. The corresponding options parameter is op_debug_config.

Arguments:

The value is the path of the .cfg configuration file. Multiple options in the configuration file are separated by commas (,).

• oom: Checks whether memory overwriting occurs in the global memory during operator execution.
  • Configuring this option retains the binary operator file (.o) and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
  • If this option is used, the following detection logic is added during operator build. You can use the dump_cce option to view the following code in the generated .cce file:

    inline __aicore__ void  CheckInvalidAccessOfDDR(xxx) {
        if (access_offset < 0 || access_offset + access_extent > ddr_size) {
            if (read_or_write == 1) {
                trap(0X5A5A0001);
            } else {
                trap(0X5A5A0002);
            }
        }
    }

• dump_cce: Retains the operator CCE file (.cce), binary operator file (.o), and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
• dump_loc: Retains the python-CCE mapping file *_loc.json, binary operator file (.o), and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
• ccec_O0: Enables the CCEC option -O0 during operator build. This option does not optimize the debugging information for later analysis of AI Core errors.
• ccec_g: Enables the CCEC option -g during operator build. This option optimizes the debugging information for later analysis of AI Core errors.
• check_flag: Checks whether pipeline synchronization signals in operators match each other during operator execution.
  • Configuring this option retains the binary operator file (.o) and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
  • If this option is used, the following detection logic is added during operator build. You can use the dump_cce option to view the following code in the generated .cce file:

      set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
      set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
      set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
      set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
      ....
      pipe_barrier(PIPE_MTE3);
      pipe_barrier(PIPE_MTE2);
      pipe_barrier(PIPE_M);
      pipe_barrier(PIPE_V);
      pipe_barrier(PIPE_MTE1);
      pipe_barrier(PIPE_ALL);
      wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
      wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
      wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
      wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
      ...

    During actual inference, if the pipeline synchronization signals in operators do not match each other, a timeout error is reported at the faulty operator, and the program is terminated. The following is an example of the error message:

    Aicore kernel execute failed, ..., fault kernel_name=operator name,...
    rtStreamSynchronizeWithTimeout execute failed....

Configuration example:

{ge::ir_option::OP_DEBUG_CONFIG, "/root/test0.cfg"}

The information about the test0.cfg file is as follows:

op_debug_config=ccec_g,oom

Restrictions:

During operator build, if you want to build only some instead of all AI Core operators, you need to add the OP_DEBUG_LIST field to the test0.cfg configuration file. By doing so, only the operators specified in the list are built, based on the options configured in OP_DEBUG_CONFIG. The OP_DEBUG_LIST field has the following requirements:

• The operator name or operator type can be specified.
• Operators are separated by commas (,). The operator type is configured in the OpType::TypeName format. The operator type and operator name can be configured in a mixed manner.
• The operator to be built must be stored in the configuration file specified by OP_DEBUG_CONFIG.

The following is a configuration example: Add the following information to the test0.cfg file:

op_debug_config=ccec_g,oom
op_debug_list=GatherV2,opType::ReduceSum

During model compilation, the GatherV2,ReduceSum operator is built based on the ccec_g and oom options.

Applicable products:

All processors

OPTION_EXPORT_COMPILE_STAT

Sets whether to generate the fusion_result.json result file of operator fusion information (including graph fusion and UB fusion) during graph build. The corresponding options parameter is ge.exportCompileStat.

This file is used to record the fusion patterns used during graph build. In this file:

• session_and_graph_id_xx_xx: thread and graph ID of the fusion result.
• graph_fusion: graph fusion.
• ub_fusion: UB fusion.

  UB fusion is not supported on the Ascend 950PR / Ascend 950DT , and therefore this information is not generated.

• match_times: number of times fusion patterns were matched during graph build.
• effect_times: number of times the matched fusion patterns actually took effect.
• repository_hit_times: number of times the UB fusion optimization repository was hit.

  UB fusion is not supported on the Ascend 950PR / Ascend 950DT , and therefore this information is not generated.

Parameter values:

• 0: The result file of operator fusion information is not generated.
• 1 (default): The result file of operator fusion information is generated when the program exits normally.
• 2: The result file of operator fusion information is generated when graph build is complete. That is, the file is generated even if the subsequent program aborts early when graph build is complete.

Configuration example:

{ge::ir_option::OPTION_EXPORT_COMPILE_STAT, "1"}

Applicable products:

All processors