Function: set_dump

Applicable Products

Product	Supported (√/x)
Ascend 950PR / Ascend 950DT	√
Atlas A3 training products / Atlas A3 inference products	√
Atlas A2 training products / Atlas A2 inference products	√
Atlas training products	√
Atlas inference products	√
Atlas 200I/500 A2 inference products	√

Description

Sets dump configuration.

Prototype

C Prototype

        
             aclError aclmdlSetDump(const char *dumpCfgPath)

Python Function
1

ret = acl.mdl.set_dump(dump_cfg_path)

Parameters

Parameter	Description
dump_cfg_path	Str, path of the configuration file, including the file name. Currently, the following dump information can be configured: (If the operator input or output contains sensitive user information, information leakage may occur.) Model dump configuration (used to export the input and output data of operators at each layer in the model) and single-operator dump configuration (used to export the input and output data of an operator). The exported data is used to compare with that of a specified model or operator to locate accuracy issues. For details about the configuration example, description, and restrictions, see Examples of Model Dump Configuration and Single-Operator Dump Configuration. This dump configuration is disabled by default. Dump configuration of the exception operator (used to export the input and output data, workspace information, and tiling information of the exception operator). The exported data is used to analyze AI Core errors. For details about the configuration example, see Example of Dump Configuration for Exception Operators. This dump configuration is disabled by default. Dump configuration of the overflow/underflow operator (used to export the input and output data of the overflow/underflow operator in the model). The exported data is used to analyze overflow/underflow causes and locate model accuracy problems. For details about the configuration example, description, and restrictions, see Example of Overflow/Underflow Operator Dump Configuration. This dump configuration is disabled by default. Operator Dump Watch mode configuration (enables the observation mode for the output data of a specified operator). If you suspect that the memory is overwritten by other operators after locating the accuracy issues of some operators and excluding the computation issues of the operators, you can enable the Dump Watch mode. For details about the configuration example and restrictions, see Dump Watch Configuration for Operators. The dump watch mode is disabled by default. This dump configuration is used to export the debugging information of the Ascend C operator kernel to locate operator problems. This dump configuration is disabled by default. This configuration is only available for the following models: Ascend 950PR / Ascend 950DT Atlas A3 training products / Atlas A3 inference products Atlas A2 training products / Atlas A2 inference products Atlas 200I/500 A2 inference products Atlas inference products

Parameter

Description

dump_cfg_path

Str, path of the configuration file, including the file name.

Currently, the following dump information can be configured: (If the operator input or output contains sensitive user information, information leakage may occur.)

Model dump configuration (used to export the input and output data of operators at each layer in the model) and single-operator dump configuration (used to export the input and output data of an operator). The exported data is used to compare with that of a specified model or operator to locate accuracy issues. For details about the configuration example, description, and restrictions, see Examples of Model Dump Configuration and Single-Operator Dump Configuration. This dump configuration is disabled by default.
Dump configuration of the exception operator (used to export the input and output data, workspace information, and tiling information of the exception operator). The exported data is used to analyze AI Core errors. For details about the configuration example, see Example of Dump Configuration for Exception Operators. This dump configuration is disabled by default.
Dump configuration of the overflow/underflow operator (used to export the input and output data of the overflow/underflow operator in the model). The exported data is used to analyze overflow/underflow causes and locate model accuracy problems. For details about the configuration example, description, and restrictions, see Example of Overflow/Underflow Operator Dump Configuration. This dump configuration is disabled by default.
Operator Dump Watch mode configuration (enables the observation mode for the output data of a specified operator). If you suspect that the memory is overwritten by other operators after locating the accuracy issues of some operators and excluding the computation issues of the operators, you can enable the Dump Watch mode. For details about the configuration example and restrictions, see Dump Watch Configuration for Operators. The dump watch mode is disabled by default.
This dump configuration is used to export the debugging information of the Ascend C operator kernel to locate operator problems. This dump configuration is disabled by default.
This configuration is only available for the following models:

Ascend 950PR / Ascend 950DT

Atlas A3 training products / Atlas A3 inference products

Atlas A2 training products / Atlas A2 inference products

Atlas 200I/500 A2 inference products

Atlas inference products

Return Values

Return Value	Description
ret	Int, error code. 0 on success; else, failure.

Restrictions

acl.mdl.init_dump must be used in pair with acl.mdl.set_dump and acl.mdl.finalize_dump to dump data to files. These APIs can be called for multiple times in a single process to obtain dump data of different Dump configurations.
Example scenario:
- To execute two models, you need to set dump information differently. The API call sequence is as follows: acl.init --> acl.mdl.init_dump --> acl.mdl.set_dump --> Model loading --> Model execution --> acl.mdl.finalize_dump --> Model unloading -> acl.mdl.init_dump -> acl.mdl.set_dump -> Model loading -> Model execution -> acl.mdl.finalize_dump -> Model unloading -> Other tasks -> acl.finalize
- To execute the same model twice, you only need to perform the dump operation for the first execution. The API call sequence is as follows: acl.init --> acl.mdl.init_dump --> acl.mdl.set_dump --> Model loading --> Model execution --> acl.mdl.finalize_dump --> Model unloading --> Model loading --> Model execution --> Other tasks --> acl.finalize

The configured dump information is valid only when the model is loaded after the dump function is enabled by calling this API. The dump configuration does not take effect on models loaded before this API call unless you reload the models after this API call.
For example, in the following API call sequence, the dump configuration is valid only for model 2:

acl.mdl.init_dump --> Model 1 loading --> acl.mdl.set_dump --> Model 2 loading --> --> acl.mdl.finalize_dump
If this API is called repeatedly to set dump configuration for the same model, the most recent configuration is applied.
For example, in the following API call sequence, the second dump configuration call overwrites the first call:

acl.mdl.init_dump --> acl.mdl.set_dump --> acl.mdl.set_dump --> Model 1 loading --> acl.mdl.finalize_dump

Reference

In addition, the acl.init API is provided. During initialization, the dump configuration is transferred through the *.json file. After the application is run, the dump data is obtained. In this mode, the acl.init API can be called only once in a process. To modify the dump configuration, modify the configuration in the *.json file.

Examples of Model Dump Configuration and Single-Operator Dump Configuration

After model dump or single-operator dump is configured, the exported data is used to compare with that of a specified model or operator to locate accuracy issues. For details about the comparison method, see Accuracy Analyzer.

Model dump configuration example:

{                                                                                            
	"dump":{
		"dump_list":[                                                                        
			{	"model_name":"ResNet-101"
			},
			{                                                                                
				"model_name":"ResNet-50",
				"layer":[
				      "conv1conv1_relu",
				      "res2a_branch2ares2a_branch2a_relu",
				      "res2a_branch1",
				      "pool1"
				] 
			}  
		],  
		"dump_path":"$HOME/output",
                "dump_mode":"output",
		"dump_op_switch":"off",
                "dump_data":"tensor"
	}                                                                                        
}

Example of single-operator dump configuration:

{
    "dump":{
        "dump_path":"output",
        "dump_list":[], 
	"dump_op_switch":"on",
        "dump_data":"tensor"
    }
}

**Table 1** Format of the **acl.json** file
Parameter	Description
dump_list	(Required) List of network-wide models for data dump. In the model inference scenario, to dump all operators, set this parameter as follows: "dump_list":[{}] When multiple models or specific operators need to be dumped, model_name and layer must be used together. In the single-operator calling scenario (including single-operator model execution and single-operator API execution), you are advised to set dump_list as follows: "dump_list":[{}]
model_name	Model name. The value of model_name of each model must be unique. To load a model from a file, enter the model file name without the name extension. You can also set this parameter to the value of the outermost name field in the .json file after ATC-based model conversion. To load a model from memory, set this parameter to the value of the name field in the .json file after ATC-based model conversion.
layer	Name of the operator to be dumped. When I/O performance is poor, full data dump may trigger execution timeouts due to excessive data volume. Full dump is therefore not recommended; specify target operators for dumping instead. The value can be the name of the operator after ATC model conversion or the name of the original operator before conversion. Configure the operator name in each line in the format. Use commas (,) to separate operators. You do not need to set model_name. In this case, the corresponding operators of all models are dumped by default. If model_name is set, the corresponding operators of the model are dumped. In the static subgraph scenario with dynamic shape, if the root graph model_name is configured, operators in both the root graph and subgraph are dumped. If the input of the specified operator involves the data operator, the data operator information is dumped. To dump the data operator, enter the downstream nodes of the data operator. To dump all operators of a model, the layer field does not need to be included.
optype_blacklist	Dump data blocklist. Input and output data of operators of the specified types in the blocklist will not be dumped, allowing users to control the volume of data to be dumped via this configuration. This function takes effect only when model data is dumped and dump_level is set to op. It can also be used together with opname_blacklist. Configuration example: { "dump":{ "dump_list":[ { "model_name":"ResNet-50", "optype_blacklist":[ { "name":"conv" "pos":["input0", "input1"] } ] } ], "dump_path":"/home/output", "dump_mode":"input", } } The preceding example indicates that the dump operation is not performed on input0 and input1 of the conv operator. conv indicates the operator type. optype_blacklist contains the name and pos fields. Pay attention to the following during configuration: name: operator type, which supports types of the operators generated after ATC model conversion. This filter rule does not take effect if left empty. pos: operator inputs or outputs. The value must be in the inputn or outputn format, where n stands for the input/output index. This filter rule does not take effect if left empty. A maximum of 100 filter entries can be configured in optype_blacklist. If model_name is specified, the configuration only applies to operators under the corresponding model. If model_name is not specified, the configuration applies to operators of all models.
opname_blacklist	Dump data blocklist. Input and output data of operators of the specified names in the blocklist will not be dumped, allowing users to control the volume of data to be dumped via this configuration. This function takes effect only when model data is dumped and dump_level is set to op. It can also be used together with optype_blacklist. Configuration example: { "dump":{ "dump_list":[ { "model_name":"ResNet-50", "opname_blacklist":[ { "name":"conv" "pos":["input0", "input1"] } ] } ], "dump_path":"/home/output", "dump_mode":"input", } } The preceding example indicates that the dump operation is not performed on input0 and input1 of the conv operator. conv indicates the operator name. opname_blacklist contains the name and pos fields. Pay attention to the following during configuration: name: operator name, which supports names of the operators generated after ATC model conversion. This filter rule does not take effect if left empty. pos: operator inputs or outputs. The value must be in the inputn or outputn format, where n stands for the input/output index. This filter rule does not take effect if left empty. A maximum of 100 filter entries can be configured in opname_blacklist. If model_name is specified, the configuration only applies to operators under the corresponding model. If model_name is not specified, the configuration applies to operators of all models.
opname_range	Dump data range. The dump operation is performed on the data within the closed interval from begin to end. This function takes effect only when model data is dumped and dump_level is set to op. Configuration example: { "dump":{ "dump_list":[ { "model_name":"ResNet-50", "opname_range":[{"begin":"conv1", "end":"relu1" }, {"begin":"conv2", "end":"pool1"}] } ], "dump_mode":"output", "dump_level": "op", "dump_path":"/home/output" } } The above example indicates that dump operations are performed on data within the closed interval from conv1 to relu1 and from conv2 to pool1, where conv1, relu1, conv2, and pool1 are operator names. Note the following during configuration: model_name cannot be left empty. The parameters in begin and end indicate operator names, which can be set to the names of operators after ATC model conversion. begin and end cannot be empty and must be non-data operators. If the inputs of operators within the range defined by begin and end involve data operators, the data operator information will also be dumped simultaneously.
dump_path	(Required) Directory for storing dump data files in the operating environment. The directory must be created in advance and the running user configured during installation must have the read and write permissions on the directory. The path can be either absolute or relative. An absolute path starting with a slash (/), for example, /home/output. A relative path starting with a directory name, for example, output.
dump_mode	Dump mode. input: dumps operator inputs only. output (default): dumps operator outputs only. all: dumps both operator inputs and outputs. Note: If this parameter is set to all, the input data of some operators, such as collective communication operators HcomAllGather and HcomAllReduce, will be modified during execution. Therefore, the system dumps the operator input before operator execution and dumps the operator output after operator execution. In this way, the dumped input and output data of the same operator is flushed to drives separately, and multiple dump files are generated. After parsing the dump files, you can determine whether the data is an input or output based on the file content.
dump_level	Dump data level. The options are as follows: op: dumps data at the operator level. kernel: dumps data at the kernel level. all (default): dumps both op and kernel level data. If the default value is used, there are a large number of dump files, for example, dump files starting with aclnn. If you have requirements on the dump performance or the memory resources are limited, you can set this parameter to the op level to improve the dump performance and reduce the number of dump files. Note: An operator represents a computational logic (such as addition, subtraction, multiplication, and division). A kernel is the implementation that executes this computational logic, and requires allocation of specific computing devices to complete computation.
dump_op_switch	Whether to enable dump data collection in the single-operator calling scenario (including single-operator model execution and single-operator API execution). on: enabled. off (default): disabled.
dump_step	Iterations to be dumped. This parameter is not required in the inference scenario. If this parameter is not configured, dump data will be generated for all iterations by default, which may result in a large amount of data. You are advised to specify iterations as required. Separate multiple iterations using vertical bars (\|), for example, 0\|5\|10. You can also use hyphens (-) to specify the iteration range, for example, 0\|3-5\|10. Configuration example: { "dump":{ "dump_list":[ ...... ], "dump_path":"/home/output", "dump_mode":"output", "dump_op_switch":"off", "dump_step": "0\|3-5\|10" } } In training scenarios, if you specify iterations for dump data collection via the dump_step parameter in acl.json, and meanwhile configure the ge.exec.dumpStep parameter (which also controls dump collection iterations) in the GEInitialize API, the last configured parameter shall take precedence. For details about the GEInitialize API, see Graph Development.
dump_data	Type of the operator dump content. The options are as follows: tensor (default): dumps operator data. stats: dumps operator statistics. The result file is in .csv format and contains the operator name, input/output data type, maximum value, and minimum value. Dumping full data typically generates large data volumes and consumes substantial time. You can first dump operator statistics, identify potentially abnormal operators based on these statistical results, and then dump only the data of those operators.
dump_stats	Type of statistical data to collect when dump_data is set to stats. Only the Atlas A2 training products / Atlas A2 inference products supports this parameter. Available values for this parameter are listed below. If no value is specified, Max, Min, Avg, Nan, Negative Inf, and Positive Inf data are collected by default. Max: maximum value in operator statistics. Min: minimum value in operator statistics. Avg: average value in operator statistics. Nan: undefined or unrepresentable values in operator statistics. This option applies only to the floating-point types half, bfloat, and float. Negative Inf: negative infinity values in operator statistics. This option applies only to the floating-point types half, bfloat, and float. Positive Inf: positive infinity values in operator statistics. This option applies only to the floating-point types half, bfloat, and float. L2norm: L2 norm value of the dump operator statistics. Configuration example: { "dump":{ "dump_list":[ ...... ], "dump_path":"/home/output", "dump_mode":"output", "dump_data":"stats", "dump_stats":["Max", "Min"] } }

Example of Dump Configuration for Exception Operators

You can enable dump for exception operators by setting dump_scene. The following is an example of the configuration file, indicating that lightweight exception dump is enabled:

{
    "dump":{
        "dump_path":"output",
        "dump_scene":"aic_err_brief_dump"
    }
}

The details are as follows:

dump_scene can be set to:
- aic_err_brief_dump: lightweight exception dump, which is used to export the input, output, and workspace data of the incorrect operators of AI Core.
- aic_err_norm_dump: common exception dump, which is used to export the shape, data type, format, and attribute information in addition to the lightweight exception dump.
- aic_err_detail_dump: exports the internal storage, register, and call stack information of AI Core in addition to the lightweight exception dump.
  When configuring this option, pay attention to the following points:
  - This option is supported only by the following models and requires the driver of 25.0.RC1 or later:
    Atlas A2 training products / Atlas A2 inference products
    
    Atlas A3 training products / Atlas A3 inference products
    
    You can click here to download the driver installation package of Ascend HDK 25.0.RC1 or later on the Firmware and Drivers page and install or upgrade the driver by referring to the document of the corresponding version.
  - During dump file export, the AI Core where an exception operator is located is suspended, which may affect the execution of other processes on the device. After dump files are exported, the AI Core is automatically restored. Therefore, you are advised not to use the aic_err_detail_dump option when multiple user service processes on the host specify the same device.
  - After the dump file is exported, the user service process on the host is forcibly exited. The error reported during the forcible exit is not used as the input for AI Core problem analysis.
  - If aic_err_detail_dump is configured and dump files are generated but not *.core files, aic_err_detail_dump is not configured successfully. In this case, aic_err_brief_dump will be used instead.
- lite_exception: light exception dump. It is provided to be compatible with earlier versions and is equivalent to aic_err_brief_dump.
dump_path is an optional parameter, indicating the path for storing exported dump files.
The priority of the dump file storage path is as follows: NPU_COLLECT_PATH environment variable > ASCEND_WORK_PATH environment variable > dump_path in the configuration file > current execution directory of the application.

For details about the environment variables, see Environment Variables.
To view the content of a dump file, convert the dump file to a NumPy file and then view the NumPy file using Python. For details about the conversion procedure, see Viewing Dump Files in Accuracy Analyzer.
If dump_scene is set to aic_err_detail_dump, you can use msDebug to view the content of an exported dump file. For details, see Operator Development Tool.
The dump configuration for exception operators cannot be enabled if the model dump configuration or single-operator dump configuration is enabled.

Example of Overflow/Underflow Operator Dump Configuration

Set dump_debug to on to enable the overflow/underflow operator configuration. The following is an example of the configuration file:

{
    "dump":{
        "dump_path":"output",
        "dump_debug":"on"
    }
}

The details are as follows:

If dump_debug is not set or set to off, the overflow/underflow operator configuration is disabled.
If the overflow/underflow operator configuration is enabled, dump_path must be set to specify the path for storing exported dump files.
After obtaining the exported data files, parse the files by referring to Collecting and Analyzing Data of Overflow/Underflow Operators in Accuracy Analyzer.
dump_path can be either absolute or relative.
- An absolute path starts with a slash (/), for example, /home.
- A relative path starts with a directory name, for example, output.
This function cannot be enabled when model or single-operator dump configuration is enabled. Otherwise, an error is returned.
Only overflow/underflow data of AI Core operators can be collected.

Dump Watch Configuration for Operators

Set dump_scene to watcher to enable the operator dump watch mode. The following is an example of the configuration file: (1) After operators A and B are executed, the output of operators C and D is dumped. (2) After the C and D operators are executed, the output of the C and D operators is also dumped. Compare the dump files of operators C and D in (1) and (2) to check whether operators A and B overwrite the output memory of operators C and D.

{
    "dump":{
        "dump_list":[
            {
                "layer":["A", "B"],
                "watcher_nodes":["C", "D"]
            }
        ],
        "dump_path":"/home/",
        "dump_mode":"output",
        "dump_scene":"watcher"
    }
}

The details are as follows:

If the operator dump watch mode is enabled, the overflow/underflow operator dump (by configuring the dump_debug parameter) or the single-operator model dump (by configuring the dump_op_switch parameter) cannot be enabled. Otherwise, an error will be reported. This mode does not take effect in the single-operator API dump scenario.
In dump_list, the layer parameter is used to configure the names of the operators that may overwrite the memory of other operators, and the watcher_nodes parameter is used to configure the names of the operators with accuracy issues possibly due to output memory being overwritten by other operators.
- If layer is specified, the output of the operators configured for watcher_nodes is dumped after all operators that support dump in the model are executed.
- If any operator in layer and watcher_nodes is not in a static graph or static subgraph, the configuration does not take effect.
- If an operator is in both layer and watcher_nodes or an operator in layer is a collective communication operator (the operator type starts with Hcom, for example, HcomAllReduce), only the dump files of operators in watcher_nodes will be exported.
- For a fused operator, use its name after fusion when you add it to watcher_nodes. Otherwise, dump files cannot be exported.
- Currently, model_name cannot be configured in dump_list.
If the operator dump watch mode is enabled, dump_path, which is the path for storing the exported dump file, must be configured.
The exported dump files cannot be viewed using a text tool. To view the content of a dump file, convert the dump file to a NumPy file and then view the NumPy file using Python. For details about the conversion procedure, see Viewing Dump Files in Accuracy Analyzer.
dump_path can be either absolute or relative.
- An absolute path starts with a slash (/), for example, /home.
- A relative path starts with a directory name, for example, output.
dump_mode is used to specify the data of the operators configured for watcher_nodes to be exported. Currently, only output can be configured.

Dump configuration of the operator kernel debugging information

Configure the dump_kernel_data parameter to enable the operator kernel debugging information dump function. The following is an example of the configuration file:

{
    "dump":{
        "dump_kernel_data":"printf,assert",
        "dump_path":"/home/"
    }
}

The details are as follows:

dump_kernel_data: specifies the type of data to be exported. Multiple types can be configured and separated by commas (,). If this field is not configured but the model dump configuration and single-operator dump configuration are enabled, the debugging information is exported based on all by default.
Available environment variables are as follows:
- all: Exports the output data of all the following types:
- printf: exports the output data debugged by AscendC::printf.
- tensor: exports the output data debugged by AscendC::DumpTensor.
- assert: exports the output data debugged by assert/ascendc_assert.
- timestamp: exports the output data debugged by AscendC::PrintTimeStamp.

dump_path: path for storing the exported dump file, which must be configured when the operator kernel debugging information dump function is enabled. The path can be an absolute path or a relative path.
The priorities of the dump file storage paths are as follows: ASCEND_DUMP_PATH environment variable > ASCEND_WORK_PATH environment variable > dump_path in the configuration file. For details about the environment variables, see Environment Variables.

The content of the exported dump file cannot be directly viewed using a text tool. To view the content, use the show_kernel_debug_data tool to parse the debugging information into a readable format. For details about how to use the tool, see show_kernel_debug_data in Ascend C Operator Development.

Parent topic: Dump Configuration