OpUT

Overview

Functions as a base class for the UT framework. It provides the following two APIs to define and run the test cases.

add_case
add_precision_case

Definition of the OpUT Test Class

Prototype
OpUT(op_type, op_module_name=None, op_func_name=None)
Parameters
- op_type: operator type.
- op_module_name: operator module name, that is, the name and path of the operator implementation file, for example, impl.add (the file path is impl/add.py). The default value is None. The argument is generated automatically based on op_type. For example, for a BiasAdd operator, the generated module name is impl.bias_add. For details about how the argument of op_module_name is generated based on op_type, see 4.
- op_func_name: operator function name in the operator implementation file. The default value is None. The argument is generated automatically based on op_type. For details about how the argument of op_func_name is generated based on op_type, see 4. For example, for an Add operator, the corresponding API must exist in impl/add.py. The API definition is as follows:
```
@check_op_params(REQUIRED_INPUT, REQUIRED_INPUT, REQUIRED_OUTPUT, KERNEL_NAME)
def add(input_x, input_y, output_z, kernel_name="add"):
```

add_case

Prototype
OpUT.add_case(support_soc=None, case=None)
Description
Adds test cases for operator build, tests whether an operator meets the related specifications, and builds an .o file.

Parameters

support_soc: used to test whether the test case file supports the Ascend AI Processor, whose value is the name of the corresponding .ini file in the $HOME/Ascend/ascend-toolkit/latestcompiler/data/platform_config directory. support_soc can be a string or a tuple or list of strings (indicating multiple SoCs are supported). If this parameter is set to all or None, all SoCs are supported.

case: This parameter is of the dict type.

The following is an example of the case of an operator without attributes:

{
    "params": [
        {
          "shape": (32, 64),
          "ori_shape": (32, 64),
          "format": "ND",
          "ori_format": (32, 64),
          "dtype": "float16"
        },
        {
          "shape": (32, 64),
          "ori_shape": (32, 64),
          "format": "ND",
          "ori_format": (32, 64),
          "dtype": "float16" 
        }
    ],
    "case_name": "test_add_case_1",
    "expect": "success"      
}

The following is an example of the case of an operator with attributes:

# Take the conv2d operator as an example. For details about operator implementation, see the conv2d operator implementation file provided in the sample project.
# def get_op_support_info(inputs, weights, bias, offset_w, outputs, strides, pads, dilations,groups=1, data_format='NCHW', offset_x=0, kernel_name="conv2d")

{
    "params": [
        # inputs (tensors)
        {'shape': (1, 1, 16, 16, 16), "format": "NC1HWC0", 'ori_shape': (1, 16, 16, 16), 'ori_format': 'NCHW', 'dtype': 'float16', "param_type": "input", "value_range": [1.0, 2.0]},
        # weights (tensors)
        {'shape': (1, 1, 16, 16), "format":"FRACTAL_Z", 'ori_shape': (16, 16, 1, 1), 'ori_format': 'NCHW', 'dtype': 'float16', "param_type": "input", "value_range": [1.0, 2.0]},
        # bias
        None,
        # offset_w
        None,
        # outputs (tensors)
        {'shape':(1, 1, 16, 16, 16), 'ori_shape':(1, 16, 16, 16), "format": "NC1HWC0",  'ori_format': 'NCHW', "param_type": "output", 'dtype': 'float16'},
        # strides
        (1, 1, 1, 1), 
        # pads
        (0, 0, 0, 0), 
        # dilations
        (1, 1, 1, 1), 
        # Optional groups
         1,
        # Optional data_format
         'NCHW', 
        # Optional offset_x
         0
    ],
    "case_name": "test_conv2d_case_1",
    "expect": "success" 
}

Table 1 describes the key fields in dict.

**Table 1** **key** field configuration
Parameter	Value
params	This field is passed through to the operator API during test case running. The parameter sequence in this field must be the same as that in the operator API. If the inputs are tensors, transfer the following fields: shape: shape of a tensor. ori_shape: original shape of a tensor. format: tensor format. ori_format: original format of a tensor. param_type: tensor type. dtype: tensor data type. value_range: tensor value range. The default value range is [0.1, 1.0]. If the inputs are not tensors, transfer the actual parameter values. If the inputs are empty, transfer None.
case_name	(Optional) name of a test case. If it is not specified, the test framework generates the argument of case_name automatically in the following format: test_{op_type}_auto_case_name_{case_count} Example: test_Add_auto_case_name_1
expect	Expected result. Defaults to success. It can also be set to RuntimeError, indicating an expected exception.

add_precision_case

Prototype
OpUT.add_precision_case(support_soc=None, case)
Description
Adds test cases for operator build and precision test.

Parameters

support_soc: used to test whether the test case file supports the Ascend AI Processor. The value is the name of the corresponding .ini file in the compiler/data/platform_config directory. support_soc can be a string or a tuple or list of strings (indicating multiple SoCs are supported). If this parameter is set to all or None, all SoCs are supported.

case: a dictionary. An example is as follows.

The following is an example of the case of an operator without attributes:

{
    "params": [
        {
          "shape": (32, 64),
          "ori_shape": (32, 64),
          "format": "ND",
          "ori_format": "ND",
          "dtype": "float16",
          "param_type": "input"
        },
        {
          "shape": (32, 64),
          "ori_shape": (32, 64),
          "format": "ND",
          "ori_format": "ND",
          "dtype": "float16",
          "param_type": "output"
        }
    ],
    "case_name": "test_add_case_1",
    "calc_expect_func": np_add # A function
    "precision_standard": precision_info.PrecisionStandard(0.001, 0.001)  # Optional field
}

The following is an example of the case of an operator with attributes:

# Take the conv2d operator as an example. For details about operator implementation, see the conv2d operator implementation file provided in the sample project.
# def get_op_support_info(inputs, weights, bias, offset_w, outputs, strides, pads, dilations,groups=1, data_format='NCHW', offset_x=0, kernel_name="conv2d")

{
    "params": [
        # inputs (tensors)
        {'shape': (1, 1, 16, 16, 16), "format": "NC1HWC0", 'ori_shape': (1, 16, 16, 16), 'ori_format': 'NCHW', 'dtype': 'float16', "param_type": "input", "value_range": [1.0, 2.0]},
        # weights (tensors)
        {'shape': (1, 1, 16, 16), "format":"FRACTAL_Z", 'ori_shape': (16, 16, 1, 1), 'ori_format': 'NCHW', 'dtype': 'float16', "param_type": "input", "value_range": [1.0, 2.0]},
        # bias
        None,
        # offset_w
        None,
        # outputs (tensors)
        {'shape':(1, 1, 16, 16, 16), 'ori_shape':(1, 16, 16, 16), "format": "NC1HWC0",  'ori_format': 'NCHW', "param_type": "output", 'dtype': 'float16'},
        # strides
        (1, 1, 1, 1), 
        # pads
        (0, 0, 0, 0), 
        # dilations
        (1, 1, 1, 1), 
        # Optional groups
         1,
        # Optional data_format
         'NCHW', 
        # Optional offset_x
         0
    "case_name": "test_conv2d_case_1",
    "calc_expect_func": np_conv2d # a function
    "precision_standard": precision_info.PrecisionStandard(0.001, 0.001)  # Optional field
}

Table 2 describes the key fields in dict.

**Table 2** **key** field configuration
Parameter	Value
params	This field is passed through to the operator API during test case running. The parameter sequence in this field must be the same as that in the operator API. If the inputs are tensors, transfer the following fields: shape: shape of a tensor. ori_shape: original shape of a tensor. format: tensor format. ori_format: original format of a tensor. param_type: tensor type. dtype: tensor data type. value_range: tensor value range. The default value range is [0.1, 1.0]. If the inputs are not tensors, transfer the actual parameter values. If the inputs are empty, transfer None.
case_name	(Optional) name of a test case. If it is not specified, the test framework generates the argument of case_name automatically in the following format: test_{op_type}_auto_case_name_{case_count} Example: test_Add_auto_case_name_1
calc_expect_func	Expected result generation function.
precision_standard	User-defined precision standard. The value can be rtol, atol, and Max_atol. rtol: relative tolerance rate. atol: absolute tolerance rate. Max_atol: (optional) maximum tolerance rate. NOTE: If this field is not set, the following default precision is used for comparison with the expected data: float16 as the data type: dual-0.1% error limit (0.001, 0.001, 0.1), that is, the error ratio is within 0.1% and the relative error is within 0.1%. float32 as the data type: dual-0.01% error limit (0.0001, 0.0001, 0.01), that is, the error ratio is within 0.01% and the relative error is within 0.01%. int8 or uint8 as the data type: (0.001, 1, 1), that is, the error ratio is within 1 and the relative error is within 0.1%.

Examples

Example 1

from op_test_frame.ut import OpUT
# ut_case is a keyword of the UT framework and cannot be modified.
ut_case = OpUT("Add", "impl.add", "add")

def np_add(x1, x2, y):
    y = (x1.get("value") + x2.get("value"), )
    return y

ut_case.add_precision_case(case={
    "params": [
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "input"
        },
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "input"
        },
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "output"
        }
    ],
    "case_name": "test_add_case_1",
    "calc_expect_func": np_add
})

if __name__ == '__main__':
    ut_case.run("Ascend910",None,"ca","/home/allan/Ascend/toolkit/tools/simulator")

ut_case.run: For details, see run API.

In the preceding example, the values of the operator inputs are not set. By default, the framework calls np.random.uniform(value_range, size=shape).astype(dtype) to automatically generate the input data for each input.

The value_range is defaulted to [0.1, 1.0]. The values of shape and dtype are consistent with those set in params. value_range can also be specified as follows:

{
    "shape": (32, 64),
    "ori_shape": (32, 64),
    "format": "ND",
    "ori_format": "ND",
    "dtype": "float16",
    "param_type": "input",
    "value_range": [2.0, 3.0]
}

You can also specify input values as follows:

{
    "shape": (32, 64),
    "ori_shape": (32, 64),
    "format": "ND",
    "ori_format": "ND",
    "dtype": "float16",
    "param_type": "input",
    "value": np.zeros((32, 64), np.float16)
}

Example 2

The following is an example of the user-defined precision standard.

from op_test_frame.common import precision_info
from op_test_frame.ut import OpUT

ut_case = OpUT("Add")
ut_case.add_precision_case(case={
    "params": [
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "input"
        },
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "input"
        },
        {
            "shape": (32, 64),
            "ori_shape": (32, 64),
            "format": "ND",
            "ori_format": "ND",
            "dtype": "float16",
            "param_type": "output"
        }
    ],
    "case_name": "test_add_case_1",
    "calc_expect_func": np_add,
    "precision_standard": precision_info.PrecisionStandard(0.1, 0.1) # Uses the standard precision of one tenth and compares it with the expected data.
})

run API

Prototype
OpUT.run(soc, case_name=None, simulator_mode=None, simulator_lib_path=None)
Description
Runs test cases.
Parameters
- soc: Ascend AI Processor to run test cases. The value is the name of the corresponding .ini file in the Ascend-CANN-Toolkit_installation_directory/ascend-toolkit/latest/compiler/data/platform_config directory. support_soc can be a string or a tuple or list of strings (indicating multiple SoCs are supported).
- case_name: the test case to run. Set this parameter to the value of case_name in add_case or add_precision_case.
- simulator_mode: The default value is None. If only the test cases of operator build are executed, you do not need to specify this parameter. When adding an accuracy test case, you need to specify the simulation mode, which can be ca or pv. If tm is specified, only the dot map is output and the accuracy is not compared.
- simulator_lib_path: path of the simulator libraries.
  The path structure is as follows:
```
simulator_lib_path/
  Ascendxxx/
      lib/
          libpv_model.so
          ...
  Ascendxxx/
      lib/
          libpv_model.so
          ...
```
If MindStudio is used to run UT cases, OpUT.run does not need to be called manually.

Parent topic: UT API Reference