Collecting and Analyzing Data of Overflow/Underflow Operators

If a floating-point exception occurs during AI service running, you can collect and parse the data of overflow/underflow operators to locate the fault.

Collecting Data of Overflowed/Underflowed Operators

In offline inference scenarios, collect overflow/underflow data by referring to aclInit in Application Development (C&C++).
In TensorFlow 1.15 training/online inference scenarios, collect overflow/underflow data by referring to More Features > Overflow/Underflow Data Collection in TensorFlow 1.15 Model Porting Guide.

Viewing Data of Overflowed/Underflowed Operators

By default, the generated overflow/underflow operator data file is stored in the {dump_path}/{time}/{device_id}/{model_name}/{model_id}/{data_index} directory, for example, /home/HwHiAiUser/output/20200808163566/0/npu_cluster_0/11/0. If no overflow/underflow data is collected, that is, no overflow occurs, the preceding directory is not generated.

The storage path and file naming rules are as follows:

dump_path: user-defined path for storing overflow/underflow data, for example, /home/HwHiAiUser/output
time: timestamp (for example, 20200808163566)
device_id: device ID.
model_name: submodel name. Multiple folders may exist at the model_name layer. If a period (.), slash (/), backslash (\), or space character appears in model_name, it is converted to an underscore (_).
model_id: subgraph ID.
data_index: iterations to detect overflow/underflow.

Two types of overflow/underflow data files are generated in the preceding directory:

The dump file of an overflow/underflow operator is generally named as: {op_type}.{op_name}.{task_id}.{stream_id}.{timestamp}. Any period (.), slash (/), backslash (\), or space in the op_type or op_name field is replaced by an underscore (_).
You can identify the name of the overflow/underflow operator based on the {op_name} field. In special scenarios (for example, single-operator API execution), the {op_name} field may not completely correspond to the operator name, but you can roughly determine the operator name and use Analyzing the Dump File of an Overflow/Underflow Operator to determine the input and output of the operator.
The dump file of an overflow/underflow operator is named as: Opdebug.Node_OpDebug.{task_id}.{stream_id}.{timestamp}.
To locate the overflow/underflow cause, follow the instructions in Analyzing the Data File of an Overflow/Underflow Operator.
- task_id is not the task ID of the overflow/underflow operator and can be ignored.
- When the command is executed in a Docker, the generated data is stored in the Docker.

Analyzing the Dump File of an Overflow/Underflow Operator

Upload the collected data files to the Toolkit installation environment.
Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann..
Run the msaccucmp.py script to convert the dump file into a NumPy file. The following is an example:
```
python3 msaccucmp.py convert -d /home/HwHiAiUser/dump -out /home/HwHiAiUser/dumptonumpy -v 2
```
The -d option enables the conversion of a single dump file or all dump files in a path.

Use Python to save the NumPy data into a .txt file. The following is an example:

        
             $ python3
>>> import numpy as np
>>> a = np.load("/home/HwHiAiUser/dumptonumpy/Pooling.pool1.1.5.1732082705016774.output.0.npy")
>>> b = a.flatten()
>>> np.savetxt("/home/HwHiAiUser/dumptonumpy/Pooling.pool1.1.5.1732082705016774.output.0.txt", b)

The shape and dtype no longer exist in the .txt file. For details, visit the NumPy website.

Analyzing the Data File of an Overflow/Underflow Operator

Since the generated overflow/underflow data is in binary format, you need to interpret the binary file into a readable format, such as JSON.

Upload the overflow/underflow data file of the operator to the Toolkit installation environment.

You are advised to go to the data_index directory with the minimum value, and use the dump file with the minimum {timestamp} for data parsing.
Go to the ${INSTALL_DIR}/tools/operator_cmp/compare directory. Replace ${INSTALL_DIR} with the CANN component directory. For example, if the installation is performed by the root user, the default file storage path is /usr/local/Ascend/cann..
Run the parse command.
```
python3 msaccucmp.py convert -d /home/HwHiAiUser/opdebug/Opdebug.Node_OpDebug.1.5.1732082705016774 -out /home/HwHiAiUser/result
```
The key options are described as follows:
- -d: directory of the overflow/underflow data file, including the file name
- -out: directory of the parsing result. If it is not specified, the current directory is used.

Find the parsed result as follows.

If both AI Core operator overflow/underflow detection and Atomic Add overflow/underflow detection are enabled, only the earliest overflow/underflow information is displayed.
In the following example, the AI Core operator overflow/underflow information appears first. Therefore, even if the Atomic Add operation has an overflow/underflow issue, its information will not be displayed.

        
         
           
           
             {
    "DHA Atomic Add": {
        "model_id": 0,
        "stream_id": 0,
        "task_id": 0,
        "task_type": 0,
        "pc_start": "0x0",
        "para_base": "0x0",
        "status": 0
    },
    "L2 Atomic Add": {
        "model_id": 0,
        "stream_id": 0,
        "task_id": 0,
        "task_type": 0,
        "pc_start": "0x0",
        "para_base": "0x0",
        "status": 0
    },
    "AI Core": {
        "model_id": 514,
        "stream_id": 563,
        "task_id": 57,
        "task_type": 0,
        "pc_start": "0x1008005b0000",
        "para_base": "0x100800297000",
        "kernel_code": "0x1008005ae000",
        "block_id": 1,
        "status": 32
    }
}

            

          

        
       

Complete field description:

The following lists all the fields that can be parsed, which may vary with actual product.

model_id: ID of the model where an overflow/underflow operator is located
stream_id: ID of the stream where an overflow/underflow operator is located
task_id: task ID of an overflow/underflow operator.
task_type: task type of an overflow/underflow operator.
context_id (reserved): context ID.
thread_id: thread ID, which is reserved.
pc_start: start of the code program of the overflow/underflow operator.
para_base: parameter start address of the overflow/underflow operator.
src_addr: communication source address in the SDMA transmission scenario.
dst_addr: communication destination address in the SDMA transmission scenario.
channel_id: channel ID.
core_id: AI Core ID.
kernel_code: start of the code program of the overflow/underflow operator, which is equivalent to pc_start.
block_id: block ID of an overflow/underflow operator.
status: status of the AI Core status register, including the overflow/underflow information. You can analyze the value of status to obtain the specific overflow/underflow error.

Analyzing the Error Cause Based on status

The status field for AI Core operator overflow/underflow detection is expressed in decimal. Convert it to hexadecimal to pinpoint specific errors.
For example, if status is 272, its hexadecimal representation is 0x00000110. It can be determined that the potential causes correspond to 0x00000010 and 0x00000100.
- 0x00000008: inversion overflow/underflow of the minimum negative sign bit of a signed integer
- 0x00000010: overflow/underflow in integer addition, subtraction, multiplication, or multiply-add operations
- 0x00000020: floating-point calculation overflow/underflow
- 0x00000080: negative input for the conversion of a floating-point number to an unsigned value
- 0x00000100: overflow/underflow during Float32-to-Float16 conversion or 32-bit signed integer-to-Float16 conversion
- 0x00000400: overflow/underflow in cube accumulation
Note: The preceding floating-point exception flags correspond to individual bits in the hexadecimal value, and multiple floating-point exceptions may occur simultaneously.
The status field for DHA Atomic Add overflow/underflow detection uses decimal values. A value greater than 0 indicates an overflow/underflow of DHA Atomic Add.

The status field for L2 Atomic Add overflow/underflow detection uses decimal values. A value greater than 0 indicates an overflow/underflow of L2 Atomic Add.

Parent topic: Extended Functions