Generating a Case Definition File

This section describes how to use the msOpST tool to generate the operator test case definition file (.json) as the input of the operator ST cases.

Obtain and edit the implementation file (.cpp) of the operator to be tested on the host.

The msOpST tool generates the ST case definition file of the operator based on the implementation file of the operator to be tested on the host. The implementation file path of the operator on the host in the operator project file is as follows.

You can click Link to obtain the operator implementation file add_custom.cpp on the host for reference.

├── framework/tf_plugin        // Directory for storing the implementation file of the operator plugin. The generation of single-operator model files does not depend on the operator plugin and can be ignored.
├── op_host                      // Implementation file on the host.
│   ├── add_custom_tiling.h    // Operator tiling definition file.
│   ├── add_custom.cpp         // Content file for operator prototype registration, shape derivation, information library, and tiling implementation.
│   ├── CMakeLists.txt
├── op_kernel                   // Implementation file on the kernel
│   ├── CMakeLists.txt   
│   ├── add_custom.cpp        // Operator implementation file

Run the following command to generate an operator test case definition file. Table 1 describes the available command-line arguments.
```
msopst create -i {operator.cpp file} -out {output path} -m {pb file} -q
```
The following is an example.
Take the AddCustom operator as an example. Run the following command:
msopst create -i Op_implementation/add_custom.cpp -out ./output
Replace Op_implementation with the path of the implementation file of the operator on the host.

After the command is executed successfully, an operator test case definition file AddCustom_case_timestamp.json is generated to the output directory of the current path.

Create an operator ST case definition file AddCustom_case.json. You can make modifications based on the following template. For details about all fields supported by the *.json file, see Table 1. For examples of test case definition files in different scenarios, see Test Case Definition File.

[
    {
        "case_name": "Test_OpType_001",
        "op": "OpType",
        "input_desc": [
            {
                "format": [],
                "type": [],
                "shape": [],
                "data_distribute": [
                    "uniform"
                ],
                "value_range": [
                    [
                        0.1,
                        1.0
                    ]
                ]
            }
        ],
        "output_desc": [
            {
                "format": [],
                "type": [],
                "shape": []
            }
        ]
    }
]

**Table 1** Operator test case definition file (.json)
Parameter		Description
case_name	-	(Required) String. Test case name
op	-	(Required) String. Operator type. Must not be empty.
error_threshold	-	(Optional) Customizes the accuracy standard. The value is a list containing two elements, formatted as "[threshold1, threshold2]". threshold1: threshold of the error between the operator output result and the benchmark data. If the actual error is greater than this threshold, the operator output result is recorded as the error data. threshold2: threshold of the ratio of error data to all data. If the actual ratio is less than this threshold, the accuracy is deemed acceptable. Otherwise, the precision does not meet the requirement. Defaults to "[0.01, 0.05]". Element value range: [0.0, 1.0]. NOTE: The configured list must be enclosed in quotation marks ("") to avoid problems (for example: -err_thr "[0.01, 0.05]"). If this parameter is set both in the JSON file of the test case and in the corresponding msOpST command, the latter value is used for comparison. If this parameter is not set in neither of them, the default accuracy standard [0.01, 0.05] set in the msOpST command is used for comparison.
st_mode	-	(Optional) String. ST mode. The value ms_python_train indicates a MindSpore operator project (only supported by the Atlas Training Series Product), and pt_python_train indicates a PyTorch operator project.
run_torch_api	-	(Optional) Sets the API for torch_api to call the operator. The value is torch.square, in which square indicates the API name. Set this parameter as needed.
expect	-	(Optional) Expected test result. It can be either of the following values. The default value is success. success: execution success. If the model conversion fails, the process will be terminated. You can view ATC logs to locate the fault. failed: execution failure. If you need to run an abnormal case, change the value of the expect field to failed. If the model conversion fails, the process continues. If the statistical results of status and expect in the ST case report are consistent, the test cases are included in "success count." Otherwise, the test cases are included in "failed count."
fuzz_impl	-	(Optional) String. Required if you need to use a fuzzing script to generate plenty of test cases. In this case, add this field and set it to *"relative or absolute path of the fuzzing script:function name"*. NOTE: You are advised not to call the fuzzing script in the directory of another user to avoid privilege escalation risks.
fuzz_case_num	-	(Optional) Int. Number of test cases generated using the fuzzing script. Required if the fuzz_impl field is added. The value ranges from 1 to 2000.
input_desc	-	(Required) Operator input description. NOTICE: The parameters in input_desc must have the same number of values. Failure to do so may result in test case generation failures. For example, if input1 supports two formats, input2 should also support two formats. The rule also applies to type, shape, data_distribute, and value_range of all inputs.
-	name	(Optional) Required for a dynamic multi-input operator. Set it in the format {InputName}{ID} based on inputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four inputs, there must be corresponding configuration for each input in input_desc and their names are xxx0, xxx1, xxx2, and xxx3, where *xxx* indicates the name of an input. Click here to see the configuration example of an operator with an uncertain number of inputs (dynamic multi-input operator).
-	format	(Required) String or 1D array of strings. Input format. Must not be empty. Common data formats are as follows: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 x W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 x W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. H0 and W0 indicate the size of a fractal, as shown in the following figure. RESERVED: reserved. If this value is used, type must be UNDEFINED, indicating that the corresponding operator input is optional. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	ori_format	(Optional) String or 1D array of strings. Set it in either of the following ways: Set it to the original format of the input. This field is required if the original operator format is not consistent with the implemented one. If this field is not configured, the original format is preserved. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
-	type	(Required) String or 1D array of strings Input data type: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float UNDEFINED: applies to an optional operator input. fuzz: automatically generates test cases in batches by using the fuzzing script. Click here to see the configuration example of an operator with the input of a complex number.
-	shape	(Required) Int: 1D or 2D array. Input shape. For static shape: The dimensions and value of shape are fixed and shape_range does not need to be configured. For dynamic shape: If the shape values contain a -1 placeholder, for example, (200, -1), the size of the second dimension is unknown. In this scenario, this parameter must be used together with shape_range to specify the value range that -1 supports. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. Left empty. Can be empty if format and type are set to UNDEFINED. Note that the configured shape must match the format.
-	ori_shape	(Optional) Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the originally defined one.
-	typical_shape	(Optional) Int, 1D or 2D array. Shape for test. If the shape field contains a -1 placeholder, add the typical_shape field in the operator test case definition file to specify a static shape value for test. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	shape_range	(Optional) Int. 1D or 2D array. Value range of shape when the operator supports dynamic shape. Defaults to [[1, -1]], indicating that shape ranges from 1 to infinity. For example, if shape is set to (200, -1) and shape_range is set to [[1, -1]], the value of the second dimension of shape ranges from 1 to infinity. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	is_const	(Optional) Bool. true: configures test cases with constant inputs. false: configures test cases with tensor inputs. Click here to see the configuration example of an operator with constant input.
-	data_distribute	(Required) String or 1D array of strings. Data distribution modes for generating test data. uniform: returns random values that are evenly distributed. normal: returns random values of the normal distribution (Gaussian distribution). beta: returns random values of Beta distribution. laplace: returns random values of Laplace distribution. triangular: returns random values of the triangular distribution. relu: returns random values that are evenly distributed and activated by the ReLU function. sigmoid: returns random values that are evenly distributed and activated by the sigmoid function. softmax: returns random values that are evenly distributed and activated by the softmax function. tanh: returns random values that are evenly distributed and activated by the tanh function. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	value_range	(Required) Int or float. 1D or 2D array. Value range. Must not be left empty. Formatted as [min_value, max_value], with min_value <= max_value. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	value	(Optional) String or tensor. Add this field to specify the input value. Two configuration methods are available: Enter the tensor data (for example, [1,2,3,4]): "value": [1,2,3,4] Specify the path of a binary file (for example, test.bin): "value": "../test.bin" Prepare the .bin file by yourself. The path can be an absolute path or a relative path of the test case definition file. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. NOTE: If the value field is added, the data_distribute and value_range fields will be ignored. In addition, ensure that the values of the format, type, and shape fields correspond to the value data, and each test case can test only one data type. Click here to see the configuration example.
output_desc	-	(Required) Operator output description. NOTICE: The number of output_desc's fields must be the same as that of input_desc's fields. Failure to do so may result in test case generation failures. For example, if inputx supports two formats, the output should also support two formats.
-	name	(Optional) String. Output name. Required for a dynamic multi-output operator. Set it in the format *{OutputName}{ID}* based on outputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four outputs, there must be corresponding configuration for each output in output_desc and their names are xxx0, xxx1, xxx2, and xxx3, where *xxx* indicates the name of an output.
-	format	(Required) String or 1D array of strings. Output format. Must not be empty. Selected from: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 x W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 x W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. H0 and W0 indicate the size of a fractal, as shown in the following figure. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	ori_format	(Optional) String or 1D array of strings. This field is required if the original format is not consistent with the original defined one. In this case, set it to the original format. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented format is the original defined one.
-	type	(Required) String, 1D array of strings, or "fuzz". Supported output data types: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float fuzz: automatically generates test cases in batches by using the fuzzing script.
-	shape	(Required) Int: 1D or 2D array. Input shape. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	ori_shape	(Optional) Int, 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the original defined one.
attr	-	Optional
-	name	Required if attr is configured. String. Attribute name. Must not be empty.
-	type	Required if attr is configured. String. Supported attribute types: bool int float string list_bool list_int list_float list_string list_list_int data_type: Set type to data_type if value of the attr field is a data type.
-	value	Required if attr is configured. Attribute value, which is type-specific. If type is set to bool, value must be true or false. If type is set to int, value must be an int. If type is set to float, value must be a floating point number. If type is set to string, value must be a string, for example, NCHW. If type is set to list_bool, value must be a list of bools, for example, [false, true]. If type is set to list_int, value must be a list of ints, for example, [1, 224, 224, 3]. If type is set to list_float, value must be a list of floats, for example, [1.0, 0.0]. If type is set to list_string, value must be a list of strings, for example, ["str1", "str2"]. If type is set to list_list_int, value must be a list of int lists, for example, [[1, 3, 5, 7], [2, 4, 6, 8]]. If type is set to data_type, value can be int8, int32, int16, int64, uint8, uint16, uint32, uint64, float, float16, float32, bool, double, complex64, complex128, or bfloat16. When value is set to fuzz, values of this parameter are automatically generated as part of the test cases generated in batches using the fuzzing script.
calc_expect_func_file	-	(Optional) String File path including the name of the function for generating expected operator data, for example, *"/home/teste/test_.py:function". Replace /home/teste/test_.py* with the implementation file of the expected operator data generation function, and *function* with the function name. NOTICE: You are advised not to call the script for generating expected data in the directory of another user to avoid privilege escalation risks.

(Optional) Customize a data generation function if you want to compare the expected operator outputs with actual outputs. For details, refer to the following steps.
1. Customize a function for generating the expected data of operator Add.
  Implement the function in Python. The file directory including the file name can be user-defined, for example, /home/test/test_add_st.py.
  
  For example, the expected operator data generation function of the Add operator is implemented as follows:
  1 2 3
  def calc_expect_func(x1, x2, y): res = x1["value"] + x2["value"] return [res, ]
  You need to create the expected data generation function of the operator based on the developed custom operator. The name of all input, output, and attribute elements in the test case definition file are used as the input parameters of the expected data generation function of the operator. If an input is optional, the default value will be specified for the input.
  
  For example, if the x3 input is optional, define the expected operator data generation function of the operator as follows:
  1
  def calc_expect_func(x1, x2, x3=None, y=None)
2. Add a comparison function to the ST case definition file, that is, OpType_xx.json. Edit the test case definition file as required.
  Add the calc_expect_func_file parameter to the operator test case definition file AddCustom_case_timestamp.json in step 2. The parameter value is /home/test/test_add_st.py:calc_expect_func.
```
[
    {
        "case_name":"Test_AddCustom_001",         
        "op": "AddCustom",                             
        "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func",   // Configure the implementation file for generating expected compute result.
        "input_desc": [...]
        ...
        ...
    }
]
```

Parent topic: msOpST (Operator Test)