Documentation
Download
文档中心
Function Debugging

Function Debugging

ge.debugDir

Sets the path for storing the debugging process files generated during operator build. The process files include but are not limited to the .o (operator binary file), .json (operator description file), and .cce files.

Files are generated in the current training script execution directory by default.

Restrictions:

  • If you want to specify the path for storing the process file of operator build, use ge.debugDir and ge.opDebugLevel together. If ge.opDebugLevel is set to 0, ge.debugDir cannot be used.
  • In addition to ge.debugDir, the ASCEND_WORK_PATH environment variable can be used to set the path for storing the debugging file generated during operator build. The priorities of the configuration methods are as follows: ge.debugDir > ASCEND_WORK_PATH > default storage path.

Mandatory/Optional: optional

Effective level: global, session, and graph

ge.exec.enable_exception_dump

Sets whether to dump data of the exception operator.
  • 0 (default): The dump function of the exception operator is disabled.
  • 1: The common exception dump (L1 exception dump) function is enabled, to dump the input and output data, tensor description (such as shape, dtype, and format), and workspace information of the exception operator.

    The dump data is stored in the following directories in descending order of priority: NPU_COLLECT_PATH > ASCEND_WORK_PATH > default directory (extra-info in the script execution directory)

  • 2: The Lite exception dump (L0 exception dump) is enabled to dump the input and output data, workspace information, and tiling information of the exception operator.

    The dump data is stored in the following directories in descending order of priority: ASCEND_WORK_PATH > default directory (/extra-info/data-dump/<device_id> in the script execution directory).

  • If the NPU_COLLECT_PATH environment variable is configured, only common exception dump information, including the input and output data of the exception operator, is collected, regardless of the value of ge.exec.enable_exception_dump. The dump data is stored in the directory specified by NPU_COLLECT_PATH.
  • L1 exception dump is common exception dump information, while L0 exception dump is lite exception dump information. Both of them export information such as the operator input and output and the workspace data. Compared with L0 exception dump, L1 exception dump provides more information. When L1 exception dump is enabled, the dtype information of each tensor is printed in the host application log file (plog), and the operator name and kernel related to the operator are also printed.

For details about the environment variables, see Environment Variables.

Configuration example: 
std::map<ge::AscendString, ge::AscendString> ge_options = {"ge.exec.enable_exception_dump", "0"},

Mandatory/Optional: optional

Effective level: global

ge.exportCompileStat

Sets whether to generate the fusion_result.json result file of operator fusion information (including graph fusion and UB fusion) during graph build.

This file records the fusion patterns used during graph build. The ge.fusionSwitchFile parameter for precision comparison can be used to disable specified fusion patterns. Disabled fusion patterns are not displayed in the fusion_result.json file. In the file:

  • session_and_graph_id_xx_xx: thread and graph ID of the fusion result.
  • graph_fusion: graph fusion.
  • ub_fusion: UB fusion.

    UB fusion is not supported on the Ascend 950PR / Ascend 950DT , and therefore this information is not generated.

  • match_times: number of times fusion patterns were matched during graph build.
  • effect_times: number of times the matched fusion patterns actually took effect.
  • repository_hit_times: number of times the UB fusion optimization repository was hit.

    UB fusion is not supported on the Ascend 950PR / Ascend 950DT , and therefore this information is not generated.

Parameter values:

  • 0: The result file of operator fusion information is not generated.
  • 1 (default): The result file of operator fusion information is generated when the program exits normally.
  • 2: The result file of operator fusion information is generated when graph build is complete. That is, the file is generated even if the subsequent program aborts early when graph build is complete.

If the ASCEND_WORK_PATH environment variable is not set, the result file is generated in the current path where the script is executed by default. If the ASCEND_WORK_PATH environment variable is set, the result file is saved in $ASCEND_WORK_PATH/FE/${Process ID}/fusion_result.json.

Configuration example:

{"ge.exportCompileStat", "1"};

Mandatory/Optional: optional

Effective level: global

ge.opDebugLevel

Sets whether to enable operator debugging. The values are as follows:

  • 0 (default): Disables operator debug. The operator build folder kernel_meta is not generated in the current execution path.
  • 1: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors.

    For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

  • 2: Enables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), and TBE instruction mapping files (operator CCE file *.cce and python-CCE mapping file *_loc.json) are generated in the folder for subsequent analysis of AI Core errors. Setting this option to 2 also disables build optimization and enables the CCE compiler debug function (the CCE compiler option is set to -O0-g).

    For the Ascend 950PR / Ascend 950DT , TBE instruction mapping files are not generated.

  • 3: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o) and operator description file (.json) are generated in the folder. You can refer to these files when analyzing operator errors.
  • 4: Disables operator debug. The kernel_meta folder is generated in the current execution path, and the operator binary file (.o), operator description file (.json), TBE instruction mapping file (operator CCE file *.cce), and UB fusion description file ({$kernel_name}_compute.json) are generated in the folder. These files can be used for problem reproduction and accuracy comparison during operator error analysis.

    For the Ascend 950PR / Ascend 950DT , neither TBE instruction mapping files nor UB fusion compute description files are generated.

  • If ge.opDebugLevel is set to 0 and op_debug_config is also set, the operator build directory kernel_meta is still generated in the current execution directory.
  • You are advised to set this parameter to 0 or 3 for training. To locate errors, set this parameter to 1 or 2, which might compromise the network performance.
  • When this option is set to 2, the CCE compiler is enabled, and the size of the operator kernel file (*.o) increases. In dynamic-shape scenarios, all possible shape cases are traversed during operator build, which may result in operator build failures caused by oversized operator kernel files. For such scenarios, you are advised not to enable the CCE compiler options.

    The following is an example error log generated when operator build fails due to oversized operator kernel files:

    message:link error ld.lld: error: InputSection too large for range extension thunk ./kernel_meta_xxxxx.o
  • When the debug function is enabled, if the model contains the following MC2 operators, the *.o, *.json, and *.cce files of these operators are not generated in the operator build folder kernel_meta.

    MatMulAllReduce

    MatMulAllReduceAddRmsNorm

    AllGatherMatMul

    MatMulReduceScatter

    AlltoAllAllGatherBatchMatMul

    BatchMatMulReduceScatterAlltoAll

Mandatory/Optional: optional

Effective level: global, session, and graph

op_debug_config

Enables or disables global memory check.

The value is the path of the .cfg configuration file. Multiple options in the configuration file are separated by commas (,).

  • oom: Checks whether memory overwriting occurs in the global memory during operator execution.
    • Configuring this option retains the binary operator file (.o) and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
    • If this option is used, the following detection logic is added during operator build. You can use the dump_cce option to view the following code in the generated .cce file: 
      inline __aicore__ void  CheckInvalidAccessOfDDR(xxx) {
    if (access_offset < 0 || access_offset + access_extent > ddr_size) {
        if (read_or_write == 1) {
            trap(0X5A5A0001);
        } else {
            trap(0X5A5A0002);
        }
    }
}
  • dump_cce: Retains the operator CCE file (.cce), binary operator file (.o), and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
  • dump_loc: Retains the python-CCE mapping file *_loc.json, binary operator file (.o), and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
  • ccec_O0: Enables the CCEC option -O0 during operator build. This option does not optimize the debugging information for later analysis of AI Core errors.
  • ccec_g: Enables the CCEC option -g during operator build. This option optimizes the debugging information for later analysis of AI Core errors.
  • check_flag: Checks whether pipeline synchronization signals in operators match each other during operator execution.
    • Configuring this option retains the binary operator file (.o) and operator description file (.json) in the kernel_meta folder under the current execution directory during operator build.
    • If this option is used, the following detection logic is added during operator build. You can use the dump_cce option to view the following code in the generated .cce file: 
        set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
  set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
  set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
  set_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
  ....
  pipe_barrier(PIPE_MTE3);
  pipe_barrier(PIPE_MTE2);
  pipe_barrier(PIPE_M);
  pipe_barrier(PIPE_V);
  pipe_barrier(PIPE_MTE1);
  pipe_barrier(PIPE_ALL);
  wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID0);
  wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID1);
  wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID2);
  wait_flag(PIPE_MTE3, PIPE_MTE2, EVENT_ID3);
  ...

      During actual inference, if the pipeline synchronization signals in operators do not match each other, a timeout error is reported at the faulty operator, and the program is terminated. The following is an example of the error message:

      Aicore kernel execute failed, ..., fault kernel_name=operator name,...
rtStreamSynchronizeWithTimeout execute failed....

Configuration example:

{"op_debug_config", "/root/test0.cfg"};

The information about the test0.cfg file is as follows:

op_debug_config = ccec_g,oom

Mandatory/Optional: optional

Effective level: global

Restrictions:

During operator build, if you want to compile only some instead of all AI Core operators, add the op_debug_list field to the test0.cfg configuration file. By doing so, only the operators specified in the list are built, based on the options configured in op_debug_config. The op_debug_list field has the following requirements:

  • The operator name or operator type can be specified.
  • Operators are separated by commas (,). The operator type is configured in the OpType::TypeName format. The operator type and operator name can be configured in a mixed manner.
  • The operator to be built must be stored in the configuration file specified by op_debug_config.

Configuration example: Add the following information to the configuration file (for example, test0.cfg) specified by op_debug_config:

op_debug_config= ccec_g,oom
op_debug_list=GatherV2,opType::ReduceSum

During model compilation, the GatherV2,ReduceSum operator is compiled based on the ccec_g and oom options.

  • When ccec compilation options (ccec_O0 and ccec_g) are enabled, the size of the operator kernel file (*.o file) increases. In dynamic shape scenarios, all possible scenarios are traversed during operator build, which may cause operator build failures due to large operator kernel files. In this case, do not enable the CCEC options.

    If the compilation failure is caused by large operator kernel files, the following log is displayed:

    message:link error ld.lld: error: InputSection too large for range extension thunk ./kernel_meta_xxxxx.o:(xxxx)

  • The ccec_O0 and oom options of the CCEC cannot be both enabled. Otherwise, an AI Core error may be reported. The following is an example of the error message: 
    ...there is an aivec error exception, core id is 49, error code = 0x4 ...
  • If the NPU_COLLECT_PATH environment variable is configured, the function of checking whether global memory overwriting occurs cannot be enabled (the configuration file specified by op_debug_config is set to oom). Otherwise, an error is reported when the compiled model file or operator kernel package is used.
  • When the build options oom, dump_cce, and dump_loc are configured, if the model contains the following MC2 operators, the *.o, *.json, and *.cce files of the operators are not generated in the operator build folder kernel_meta.

    MatMulAllReduce

    MatMulAllReduceAddRmsNorm

    AllGatherMatMul

    MatMulReduceScatter

    AlltoAllAllGatherBatchMatMul

    BatchMatMulReduceScatterAlltoAll

Parent topic: Options
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details