Preparing Dump Data Files of an Offline Model

Precautions

Before dumping data, build and run the application project of the model to ensure that the project is normal.
Dump data is generated during each inference. When the number of iterations is large, the amount of dump data increases accordingly. Therefore, you are advised to perform only one inference when dumping data. For LLMs, dump data is typically voluminous and time-consuming. You can enable operator statistics using dump_data, identify potentially abnormal operators based on the statistics, and then dump only those operators.
In Docker scenarios, dump is not supported in containers.
The aclInit() and aclmdlSetDump() APIs are provided to dump data.
- For details about how to use the aclInit() API, see aclInit.
- For details about how to use the aclmdlSetDump() API, see aclmdlSetDump.

Dump Data Generation

Perform the following steps to dump data of the offline model:

Open the code file of the inference application project where the aclInit() function is located, view the called aclInit() or aclmdlSetDump() function, and obtain the path of the acl.json file.

If aclInit() or aclmdlSetDump() is initialized to empty, pass the acl.json path created in 2 to the call. The acl.json path is relative to the path of the binary file generated during project build.

Modify the acl.json file in the directory (if the file does not exist, create it in the out directory after project build) and add the dump configuration in the following format.

In the model inference scenario, enable dump data collection.

         
          
            
            
              {                                                                                            
	"dump":{
		"dump_list":[                                                                        
			{	"model_name":"ResNet-101"
			},
			{                                                                                
				"model_name":"ResNet-50",
				"layer":[
				      "conv1conv1_relu",
				      "res2a_branch2ares2a_branch2a_relu",
				      "res2a_branch1",
				      "pool1"
				] 
			}  
		],  
		"dump_path":"/home/output",
                "dump_mode":"output",
		"dump_op_switch":"off",
                "dump_data":"tensor"
	}                                                                                        
}

             

           

         
        

In the single-operator calling scenario, enable dump data collection.

        
             {
    "dump":{
        "dump_path":"/home/output",
        "dump_list":[{}], 
	"dump_op_switch":"on",
        "dump_data":"tensor"
    }
}

Table 1 Format of the acl.json file

Parameter

Description

dump_list

(Required) List of network-wide models for data dump.

In the model inference scenario, to dump all operators, set this parameter as follows:
```
"dump_list":[{}]
```
When multiple models or specific operators need to be dumped, model_name and layer must be used together.
In the single-operator calling scenario (including single-operator model execution and single-operator API execution), you are advised to set dump_list as follows:
```
"dump_list":[{}]
```

model_name

Model name. The value of model_name of each model must be unique.

To load a model from a file, enter the model file name without the name extension. You can also set this parameter to the value of the outermost name field in the .json file after ATC-based model conversion.
To load a model from memory, set this parameter to the value of the name field in the .json file after ATC-based model conversion.

layer

Name of the operator to be dumped. When I/O performance is poor, full data dump may trigger execution timeouts due to excessive data volume. Full dump is therefore not recommended; specify target operators for dumping instead. The value can be the name of the operator after ATC model conversion or the name of the original operator before conversion.

Configure the operator name in each line in the format. Use commas (,) to separate operators.
You do not need to set model_name. In this case, the corresponding operators of all models are dumped by default. If model_name is set, the corresponding operators of the model are dumped.
If the input of the specified operator involves the data operator, the data operator information is dumped. To dump the data operator, enter the downstream nodes of the data operator.
To dump all operators of a model, the layer field does not need to be included.

optype_blacklist

Dump data blocklist. Input and output data of operators of the specified types in the blocklist will not be dumped, allowing users to control the volume of data to be dumped via this configuration.

This function takes effect only when model data is dumped and dump_level is set to op. It can also be used together with opname_blacklist.

Configuration example:

{
	"dump":{
		"dump_list":[     
			{                                                                                
				"model_name":"ResNet-50",
				"optype_blacklist":[
				    {
					  "name":"conv"
					  "pos":["input0", "input1"]
					} 
				] 
			}
		],  
		"dump_path":"/home/output",
                "dump_mode":"input",
	}  
}

The preceding example indicates that the dump operation is not performed on input0 and input1 of the conv operator. conv indicates the operator type.

optype_blacklist contains the name and pos fields. Pay attention to the following during configuration:

name: operator type, which supports types of the operators generated after ATC model conversion. This filter rule does not take effect if left empty.
pos: operator inputs or outputs. The value must be in the inputn or outputn format, where n stands for the input/output index. This filter rule does not take effect if left empty.
A maximum of 100 filter entries can be configured in optype_blacklist.
If model_name is specified, the configuration only applies to operators under the corresponding model. If model_name is not specified, the configuration applies to operators of all models.

opname_blacklist

Dump data blocklist. Input and output data of operators of the specified names in the blocklist will not be dumped, allowing users to control the volume of data to be dumped via this configuration.

This function takes effect only when model data is dumped and dump_level is set to op. It can also be used together with optype_blacklist.

Configuration example:

{
	"dump":{
		"dump_list":[     
			{                                                                                
				"model_name":"ResNet-50",
				"opname_blacklist":[
				    {
					  "name":"conv"
					  "pos":["input0", "input1"]
					} 
				] 
			}
		],  
		"dump_path":"/home/output",
                "dump_mode":"input",
	}  
}

The preceding example indicates that the dump operation is not performed on input0 and input1 of the conv operator. conv indicates the operator name.

opname_blacklist contains the name and pos fields. Pay attention to the following during configuration:

name: operator name, which supports names of the operators generated after ATC model conversion. This filter rule does not take effect if left empty.
pos: operator inputs or outputs. The value must be in the inputn or outputn format, where n stands for the input/output index. This filter rule does not take effect if left empty.
A maximum of 100 filter entries can be configured in opname_blacklist.
If model_name is specified, the configuration only applies to operators under the corresponding model. If model_name is not specified, the configuration applies to operators of all models.

opname_range

Dump data range. The dump operation is performed on the data within the closed interval from begin to end.

This function takes effect only when model data is dumped and dump_level is set to op.

Configuration example:

{
	"dump":{
		"dump_list":[
			{
				"model_name":"ResNet-50",
				"opname_range":[{"begin":"conv1", "end":"relu1" }, {"begin":"conv2", "end":"pool1"}]
			}
		],
		"dump_mode":"output",
        "dump_level": "op",
        "dump_path":"/home/output"
	}
}

The above example indicates that dump operations are performed on data within the closed interval from conv1 to relu1 and from conv2 to pool1, where conv1, relu1, conv2, and pool1 are operator names.

Note the following during configuration:

model_name cannot be left empty.
The parameters in begin and end indicate operator names, which can be set to the names of operators after ATC model conversion.
begin and end cannot be empty and must be non-data operators. If the inputs of operators within the range defined by begin and end involve data operators, the data operator information will also be dumped simultaneously.

dump_path

(Required) Directory for storing dump data files in the operating environment. The directory must be created in advance and the running user configured during installation must have the read and write permissions on the directory.

The path can be either absolute or relative.

An absolute path starting with a slash (/), for example, /home/output.
A relative path starting with a directory name, for example, output.

dump_mode

Dump mode.

input: dumps operator inputs only.
output (default): dumps operator outputs only.
all: dumps both operator inputs and outputs.
Note: If this parameter is set to all, the input data of some operators, such as collective communication operators HcomAllGather and HcomAllReduce, will be modified during execution. Therefore, the system dumps the operator input before operator execution and dumps the operator output after operator execution. In this way, the dumped input and output data of the same operator is flushed to drives separately, and multiple dump files are generated. After parsing the dump files, you can determine whether the data is an input or output based on the file content.

dump_level

Dump data level. The options are as follows:

op: dumps data at the operator level.
kernel: dumps data at the kernel level.
all (default): dumps both op and kernel level data.

If the default value is used, there are a large number of dump files, for example, dump files starting with aclnn. If you have requirements on the dump performance or the memory resources are limited, you can set this parameter to the op level to improve the dump performance and reduce the number of dump files.

Note: An operator represents a computational logic (such as addition, subtraction, multiplication, and division). A kernel is the implementation that executes this computational logic, and requires allocation of specific computing devices to complete computation.

dump_op_switch

Whether to enable dump data collection in the single-operator calling scenario (including single-operator model execution and single-operator API execution).

on: enabled.
off (default): disabled.

dump_step

Iterations to be dumped. This parameter is not required in the inference scenario.

If this parameter is not configured, dump data will be generated for all iterations by default, which may result in a large amount of data. You are advised to specify iterations as required.

Separate multiple iterations using vertical bars (|), for example, 0|5|10. You can also use hyphens (-) to specify the iteration range, for example, 0|3-5|10.

Configuration example:

              
                   {
	"dump":{
		"dump_list":[     
			...... 
		],  
		"dump_path":"/home/output",
                "dump_mode":"output",
		"dump_op_switch":"off",
                "dump_step": "0|3-5|10"
	}  
}

In training scenarios, if you specify iterations for dump data collection via the dump_step parameter in acl.json, and meanwhile configure the ge.exec.dumpStep parameter (which also controls dump collection iterations) in the GEInitialize API, the last configured parameter shall take precedence. For details about the GEInitialize API, see Graph Development.

dump_data

Type of the operator dump content. The options are as follows:

tensor (default): dumps operator data.
stats: dumps operator statistics. The result file is in .csv format and contains the operator name, input/output data type, maximum value, and minimum value.

Dumping full data typically generates large data volumes and consumes substantial time. You can first dump operator statistics, identify potentially abnormal operators based on these statistical results, and then dump only the data of those operators.

dump_stats

Type of statistical data to collect when dump_data is set to stats.

Only the Atlas A2 training product / Atlas A2 inference product supports this parameter.

Available values for this parameter are listed below. If no value is specified, Max, Min, Avg, Nan, Negative Inf, and Positive Inf data are collected by default.

Max: maximum value in operator statistics.
Min: minimum value in operator statistics.
Avg: average value in operator statistics.
Nan: undefined or unrepresentable values in operator statistics. This option applies only to the floating-point types half, bfloat, and float.
Negative Inf: negative infinity values in operator statistics. This option applies only to the floating-point types half, bfloat, and float.
Positive Inf: positive infinity values in operator statistics. This option applies only to the floating-point types half, bfloat, and float.
L2norm: L2 norm value in operator statistics.

Configuration example:

              
                   {
    "dump":{
	"dump_list":[     
		...... 
	],  
        "dump_path":"/home/output",
        "dump_mode":"output",
        "dump_data":"stats",
        "dump_stats":["Max", "Min"]
    }
}

Run the application to generate dump data files. The path and format of the generated dump data files are described as follows.

In model inference scenarios, the disk storage path for dumped data is as follows: {dump_path}/{time}/{device_id}/{model_name}/{model_id}/{data_index}/{dump_file}.

In single-operator calling scenarios (including single-operator model execution and single-operator API execution), the disk storage path for dumped data is as follows: {dump_path}/{time}/{device_id}/{dump_file}.

**Table 2** Path format of a dump file
Path Key	Description	Note
dump_path	Dump path configured in the acl.json file.	The dump data file is named in the {op_type}.{op_name}.{task_id}.{stream_id}.{timestamp} format.
time	Dump time.	Formatted as YYYYMMDDHHMMSS.
device_id	Device ID.	-
model_name	Model name.	Periods (.), forward slashes (/), backslashes (\), and spaces in model_name are replaced with underscores (_).
model_id	Model ID.	-
data_index	Execution sequence number of each task, indexed starting at 0. This value is increased by 1 every dump.	-

Periods (.), forward slashes (/), backslashes (\), and spaces in op_type or op_name in the dump file are replaced with underscores (_).
If the filename exceeds the OS filename length limit (typically 255 characters), the dump file will be renamed to a string of random numbers. You can check the mapping.csv file in the same directory for the name mapping relationship.
No dump data will be generated for the following operators during graph execution:
- Operators confirmed not to execute on the device before graph execution, including conditional operators (such as if, while, for, and case), data operators (such as Data, RefData, and Const), and data flow operators (such as StackPush, StackPop, Concat, and Split).
- Operators marked by the GE during graph optimization to skip execution on the device. For such operators, the _no_task attribute of attr in the dump graph is true.
- Operators located on unreachable execution branches in the graph.

Parent topic: Comparison Between NPU and NPU (Offline Inference)