aclmdlSetDump

Applicable Products

Product	Supported
Ascend 950PR / Ascend 950DT	√
Atlas A3 training products / Atlas A3 inference products	√
Atlas A2 training products / Atlas A2 inference products	√
Atlas 200I/500 A2 inference products	√
Atlas inference products	√
Atlas training products	√

Function

Sets dump parameters.

aclmdlInitDump, aclmdlSetDump, and aclmdlFinalizeDump work together to record dump data to files. These APIs can be called multiple times in a single process to obtain dump data of different dump configurations. The specific scenarios are exemplified as follows:

To execute two different models, you need to set dump configurations differently. The API call sequence is as follows: aclInit --> aclmdlInitDump --> aclmdlSetDump --> model 1 loading --> model 1 execution --> aclmdlFinalizeDump --> model 1 unloading --> aclmdlInitDump --> aclmdlSetDump --> model 2 loading --> model 2 execution --> aclmdlFinalizeDump --> model 2 unloading --> execution of other tasks --> aclFinalize
To execute the same model twice, you only need to perform the dump operation for the first execution. The API call sequence is as follows: aclInit --> aclmdlInitDump --> aclmdlSetDump --> model loading --> model execution --> aclmdlFinalizeDump --> model unloading --> model loading --> model execution --> execution of other tasks --> aclFinalize

Prototype

      
           aclError aclmdlSetDump(const char *dumpCfgPath)

Parameters

Parameter	Input/Output	Description
dumpCfgPath	Input	Pointer to the configuration file path, including the file name. The configuration file is in JSON format. You can use this configuration file to enable or configure various dump information. For details, see the description in the following function configuration examples. If the operator input or output contains sensitive user information, there may be risks of information leakage.

Parameter

Input/Output

Description

dumpCfgPath

Input

Pointer to the configuration file path, including the file name. The configuration file is in JSON format.

You can use this configuration file to enable or configure various dump information. For details, see the description in the following function configuration examples. If the operator input or output contains sensitive user information, there may be risks of information leakage.

Returns

0 on success; otherwise, failure. For details, see aclError.

Restrictions

The configured dump information is valid only when the model is loaded after the dump function is enabled by calling this API. The dump configuration does not take effect on models loaded before this API call unless you reload the models after this API call.
For example, in the following API call sequence, the dump configuration is valid only for model 2:

aclmdlInitDump --> model 1 loading --> aclmdlSetDump --> model 2 loading --> aclmdlFinalizeDump
If this API is called repeatedly to set the dump configuration for the same model, the most recent configuration is applied.
For example, in the following API call sequence, the dump configuration of the later call overwrites that of the previous call.

aclmdlInitDump --> aclmdlSetDump --> aclmdlSetDump --> model 1 loading --> aclmdlFinalizeDump

Model Dump Configuration and Single-Operator Dump Configuration

Model dump configuration is used to export the input and output data of operators at each layer in a model. Single-operator dump configuration is used to export the input and output data of a single operator. The exported data is used to compare with that of a specified model or operator to locate accuracy issues. For details about configuration examples, see Accuracy Analyzer. This dump configuration is disabled by default.

To enable the dump configuration through this API, you need to use the dump_path parameter to configure the path for storing dump data. For details about the parameter, see Preparing Dump Data of an Offline Model in Accuracy Analyzer.

Example of model dump configuration:

      
       
         
         
           {                                                                                            
	"dump":{
		"dump_list":[                                                                        
			{	"model_name":"ResNet-101"
			},
			{                                                                                
				"model_name":"ResNet-50",
				"layer":[
				      "conv1conv1_relu",
				      "res2a_branch2ares2a_branch2a_relu",
				      "res2a_branch1",
				      "pool1"
				] 
			}  
		],  
		"dump_path":"/home/output",
                "dump_mode":"output",
		"dump_op_switch":"off",
                "dump_data":"tensor"
	}                                                                                        
}

          

        

      
     

Example of single-operator dump configuration:

      
           {
    "dump":{
        "dump_path":"/home/output",
        "dump_list":[{}], 
	"dump_op_switch":"on",
        "dump_data":"tensor"
    }
}

Configuration File Example (Exception Operator Dump Configuration)

Exception operator dump configuration (used to export the input and output data, workspace information, and tiling information of the exception operator). The exported data is used to analyze AI Core errors. Dump configurations are disabled by default.

You can enable dump for exception operators by setting dump_scene. The following is an example of the configuration file, indicating that lightweight exception dump is enabled:

{
    "dump":{
        "dump_path":"output",
        "dump_scene":"aic_err_brief_dump"
    }
}

The details are as follows:

dump_scene can be set to:
- aic_err_brief_dump: lightweight exception dump, which is used to export the input, output, and workspace data of exception operators of AI Core.
- aic_err_norm_dump: common exception dump, which is used to export the shape, data type, format, and attribute information in addition to the lightweight exception dump.
- aic_err_detail_dump: exports the internal storage, register, and call stack information of AI Core in addition to the lightweight exception dump.
  When configuring this parameter, note that:
  - This parameter is only available for the following models and requires the driver of 25.0.RC1 or later:
    Atlas A2 training products / Atlas A2 inference products
    
    Atlas A3 training products / Atlas A3 inference products
    
    You can click here to download the driver installation package of Ascend HDK 25.0.RC1 or later on the Firmware and Drivers page and install or upgrade the driver by referring to the document of the corresponding version.
  - If the parameter is set to aic_err_detail_dump, this API must be called before the aclrtSetDevice call. In addition, aclmdlFinalizeDump cannot be used to deinitialize the dump.
  - During dump file export, the AI Core running the faulty operator will be suspended, which may disrupt normal execution of other service processes on the device. The AI Core will automatically resume operation after dump files are exported.
  - Once dump files are exported, the host-side user service process will be forcibly terminated. Any error logs generated during this forced termination shall not be used for AI Core fault analysis.
  - If multiple user service processes on the host are specified with the same device and are configured with aic_err_detail_dump, the processes that are executed first export the dump files based on aic_err_detail_dump, and the processes that are executed later export the dump files based on aic_err_brief_dump.
- lite_exception: light exception dump, designed for compatibility with earlier versions and equivalent to aic_err_brief_dump.
dump_path is an optional parameter, indicating the path for storing exported dump files.
The priority of the dump file storage path is as follows: NPU_COLLECT_PATH environment variable > ASCEND_WORK_PATH environment variable > dump_path in the configuration file > current execution directory of the app.

For details about environment variables, see Environment Variables.
For details about how to view the content of an exported dump file, see Parsing Dump Files in Troubleshooting.
If dump_scene is set to aic_err_detail_dump, you can use msDebug to view the content of an exported dump file. For details, see Operator Development Tool.
The dump configuration for exception operators cannot be enabled if the model dump configuration or single-operator dump configuration is enabled.

Dump Configuration for Overflow/Underflow Operators

This configuration is used to export the input and output data of overflow/underflow operators in a model. The exported data is used to analyze the cause of overflow/underflow and locate model accuracy issues. This dump configuration is disabled by default.

If dump_debug is set to on, the overflow/underflow operator configuration is enabled. The following gives a configuration example:

{
    "dump":{
        "dump_path":"output",
        "dump_debug":"on"
    }
}

The details are as follows:

If dump_debug is not set or set to off, the overflow/underflow operator configuration is disabled.
If the overflow/underflow operator configuration is enabled, dump_path must be set to specify the path for storing exported dump files.
After obtaining the exported data files, parse the files by referring to Collecting and Analyzing Data of Overflow/Underflow Operators in Accuracy Analyzer.
dump_path can be either absolute or relative.
- An absolute path starts with a slash (/), for example, /home.
- A relative path starts with a directory name, for example, output.
The overflow/underflow operator configuration cannot be enabled if the model dump configuration or single-operator dump configuration is enabled. Otherwise, an error is returned.
Only overflow/underflow data of AI Core operators can be collected.

Dump Watch Configuration for Operators

This configuration is used to enable the watch mode for the output data of a specified operator. After locating the accuracy issues of some operators and excluding the computation issues of the operators, you can enable the dump watch mode if you suspect that the accuracy issues are caused by memory overwriting by other operators. The dump watch mode is disabled by default.

Set dump_scene to watcher to enable dump watch for operators. Below is an example of the content in the configuration file. The configuration effect is as follows: (1) After operators A and B are executed, the output of operators C and D is dumped; (2) After operators C and D are executed, the output of operators C and D is also dumped. The dump files of operators C and D in (1) will be compared with those in (2) to check whether operator A or B overwrites the output memory of operator C or D.

{
    "dump":{
        "dump_list":[
            {
                "layer":["A", "B"],
                "watcher_nodes":["C", "D"]
            }
        ],
        "dump_path":"/home/",
        "dump_mode":"output",
        "dump_level":"op",
        "dump_scene":"watcher"
    }
}

The details are as follows:

Dump watch cannot be applied in the single-operator API dump scenario.
If dump watch is enabled for operators, the overflow/underflow operator dump (by setting dump_debug) and single-operator dump (by setting dump_op_switch) cannot be enabled. Otherwise, an error will be reported.
In dump_list, layer is the operators that may overwrite the memory of other operators, and watcher_nodes is the operators whose output memory may be overwritten by other operators. If the output of an operator is overwritten, the operator accuracy may decrease.
- If layer is not specified, the output of operators listed in watcher_nodes will be dumped after all operators that support dump in the model are executed.
- If any operator in layer and watcher_nodes is not in a static graph or static subgraph, the configuration does not take effect.
- If an operator is in both layer and watcher_nodes or an operator in layer is a collective communication operator (the operator type starts with Hcom, for example, HcomAllReduce), only the dump files of operators in watcher_nodes will be exported.
- For a fused operator, use its name after fusion when you add it to watcher_nodes. Otherwise, dump files cannot be exported.
- Currently, model_name cannot be configured in dump_list.
If the dump watch configuration is enabled for operators, dump_path must be set to specify the path for storing exported dump files.
The exported dump files cannot be viewed using a text tool. To view the content of a dump file, convert the dump file to a NumPy file and then view the NumPy file using Python. For details about the conversion procedure, see Viewing Dump Files in Accuracy Analyzer.
dump_path can be either absolute or relative.
- An absolute path starts with a slash (/), for example, /home.
- A relative path starts with a directory name, for example, output.
dump_mode is used to control what data of operators in watcher_nodes can be exported. Currently, the value can only be output.
dump_level is used to set the level of dump data. The value can be:
- op: dumps data at the operator level.
- kernel: dumps data at the kernel level.
- all (default): dumps data at both the op level and the kernel level.
If the default value is used, there are a large number of dump files, for example, dump files starting with aclnn. If you have requirements on the dump performance or the memory resources are limited, you can set this parameter to the op level to improve the dump performance and reduce the number of dump files.

Note: An operator represents a computational logic (such as addition, subtraction, multiplication, and division). A kernel is the implementation that executes this computational logic, and requires allocation of specific computing devices to complete computation.

Dump Configuration for Operator Kernel Debugging Information

This dump configuration is used to export the debugging information of the Ascend C operator kernel to locate operator problems. This dump configuration is disabled by default.

This configuration is only available for the following models:

Ascend 950PR / Ascend 950DT

Atlas A3 training products / Atlas A3 inference products

Atlas A2 training products / Atlas A2 inference products

Atlas 200I/500 A2 inference products

Atlas inference products

Set the dump_kernel_data parameter to enable the dump of operator kernel debugging information. The following gives a configuration example:

{
    "dump":{
        "dump_kernel_data":"printf,assert",
        "dump_path":"/home/"
    }
}

The details are as follows:

dump_kernel_data: specifies the type of data to be exported. Multiple types can be configured and are separated by commas (,). If this field is not configured but the model dump configuration and single-operator dump configuration are enabled, all debugging information is exported by default.
Currently, the value can be:
- all: exports the output data of all types.
- printf: exports the output data generated through AscendC::printf.
- tensor: exports the output data generated through AscendC::DumpTensor.
- assert: exports the output data generated through assert/ascendc_assert.
- timestamp: exports the output data generated through AscendC::PrintTimeStamp.

dump_path: specifies the path for storing dump files. This parameter is mandatory if dump is enabled for operator kernel debugging information. The path can be an absolute path or a relative path.
The priorities of the dump file storage paths are as follows: Environment variable ASCEND_DUMP_PATH > Environment variable ASCEND_WORK_PATH > dump_path in the configuration file. For details about the environment variables, see Environment Variables.

The exported dump files cannot be viewed using a text tool. To view the content of a dump file, use the show_kernel_debug_data tool to parse the debugging information into a readable format. For details about how to use the tool, see show_kernel_debug_data in Ascend C Operator Development.

Reference

Currently, the aclInit API is also provided. During initialization, the dump configuration is passed as a JSON configuration file to dump the app data at runtime. In this mode, aclInit can be called only once in a process. To change the dump configuration, modify the JSON configuration file.

Parent topic: Dump Configuration