Generating a Case Definition File

This section describes how to use the msOpST tool to generate the operator test case definition file (*.json) as the input of the operator ST cases.

Obtain and edit the implementation file (.cpp file) of the operator to be tested on the host.

The msOpST tool generates the ST case definition file of the operator based on the implementation file of the operator to be tested on the host. The following describes the implementation file path of the operator on the host in the operator project file.

You can click link to obtain the operator implementation file add_custom.cpp on the host for reference.

This example project does not support the Atlas A3 Training Series Product.

├── framework/tf_plugin        // Directory for storing the implementation file of the operator plugin. The generation of single-operator model files does not depend on the operator plugin and can be ignored.
├── op_host                      // Implementation file on the host.
│   ├── add_custom_tiling.h    // Operator tiling definition file.
│   ├── add_custom.cpp         // Content file for operator prototype registration, shape derivation, information library, and tiling implementation.
│   ├── CMakeLists.txt
├── op_kernel                   // Implementation file on the kernel
│   ├── CMakeLists.txt   
│   ├── add_custom.cpp        // Operator implementation file

Run the following command to generate an operator test case definition file. Table 1 describes the available command-line arguments.
```
msopst create -i {operator.cpp file} -out {output path} -m {pb file} -q
```
The following is an example.
Take the AddCustom operator as an example. Run the following command:
msopst create -i Op_implementation/add_custom.cpp -out ./output
Replace Op_implementation with the path of the implementation file of the operator on the host.

After the command is executed successfully, an operator test case definition file AddCustom_case_timestamp.json is generated to the output directory of the current path.

Create an operator ST case definition file AddCustom_case.json. You can make modifications based on the following template. For details about all fields supported by the *.json file, see Table 1. For examples of test case definition files in different scenarios, see Test Case Definition File.

[
    {
        "case_name": "Test_OpType_001",
        "op": "OpType",
        "input_desc": [
            {
                "format": [],
                "type": [],
                "shape": [],
                "data_distribute": [
                    "uniform"
                ],
                "value_range": [
                    [
                        0.1,
                        1.0
                    ]
                ]
            }
        ],
        "output_desc": [
            {
                "format": [],
                "type": [],
                "shape": []
            }
        ]
    }
]

**Table 1** Operator test case definition file (.json)
Parameter		Description
case_name	-	(Required) String. Test case name
op	-	(Required) String. Operator type. Must not be empty.
error_threshold	-	Optional. Customizes the accuracy standard. The value is a list containing two elements, formatted as "[threshold1, threshold2]". threshold1: threshold of the error between the operator output result and the benchmark data. If the actual error is greater than this threshold, the operator output result is recorded as the error data. threshold2: threshold of the ratio of error data to all data. If the actual ratio is less than this threshold, the accuracy is deemed acceptable. Otherwise, the precision does not meet the requirement. Defaults to "[0.01, 0.05]". Element value range: [0.0, 1.0]. NOTE: The configured list must be enclosed in quotation marks ("") to avoid problems (for example: -err_thr "[0.01, 0.05]"). If this parameter is set both in the JSON file of the test case and in the corresponding msOpST command, the latter value is used for comparison. If this parameter is not set in neither of them, the default accuracy standard [0.01, 0.05] set in the msOpST command is used for comparison.
st_mode	-	This function is optional. A string. ST mode. The value ms_python_train indicates a MindSpore operator project (only supported by the Atlas training products), and pt_python_train indicates a PyTorch operator project.
run_torch_api	-	This function is optional. Sets the API for torch_api to call the operator. The value is torch.square, in which square indicates the API name. Set this parameter as needed.
expect	-	This function is optional. Expected test result. It can be either of the following values. The default value is success. success: execution success. If the model conversion fails, the process will be terminated. You can view ATC logs to locate the fault. failed: execution failure. If you need to run an abnormal case, change the value of the expect field to failed. If the model conversion fails, the process continues. If the statistical results of status and expect in the ST case report are consistent, the test cases are included in "success count." Otherwise, the test cases are included in "failed count."
fuzz_impl	-	(Optional) String. Required if you need to use a fuzzing script to generate plenty of test cases. In this case, add this field and set it to *"relative or absolute path of the fuzzing script:function name"*. NOTE: You are advised not to call the fuzzing script in the directory of another user to avoid privilege escalation risks.
fuzz_case_num	-	Optional. An int. Number of test cases generated using the fuzzing script. Required if the fuzz_impl field is added. The value ranges from 1 to 2000.
input_desc	-	(Required) Operator input description. NOTICE: The parameters in input_desc must have the same number of values. Failure to do so may result in test case generation failures. For example, if input1 supports two formats, input2 should also support two formats. The rule also applies to type, shape, data_distribute, and value_range of all inputs.
-	name	This function is optional. Required for a dynamic multi-input operator. Set it in the format {InputName}{ID} based on inputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four inputs, there must be corresponding configuration for each input in input_desc and their names are xxx0, xxx1, xxx2, and xxx3, where *xxx* indicates the name of an input. Click here to see the configuration example of an operator with an uncertain number of inputs (dynamic multi-input operator).
-	format	Required. A string or a 1D array of strings. Input layout. Must not be empty. Common data layouts are as follows: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 × W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 × W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 x W0) indicates the size of a fractal, as shown in the following figure. RESERVED: reserved. If this value is used, type must be UNDEFINED, indicating that the corresponding operator input is optional. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	ori_format	This function is optional. String or 1D array of strings. Set it in either of the following ways: Set it to the original format of the input. This field is required if the original operator format is not consistent with the implemented one. If this field is not configured, the original format is preserved. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
-	type	Required. String or 1D array of strings Input data type: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float bfloat16 (supported only by Atlas A3 training products/Atlas A3 inference products and Atlas A2 Training Series Product/Atlas 800I A2 Inference Product) UNDEFINED: applies to an optional operator input. fuzz: automatically generates test cases in batches by using the fuzzing script. Click here to see the configuration example of an operator with the input of a complex number.
-	shape	Required. Int: 1D or 2D array. Input shape. For static shape: The dimensions and value of shape are fixed and shape_range does not need to be configured. For dynamic shape: If the shape values contain a -1 placeholder, for example, (200, -1), the size of the second dimension is unknown. In this scenario, this parameter must be used together with shape_range to specify the value range that -1 supports. String: "fuzz". Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. Left empty. Can be empty if format and type are set to UNDEFINED. Note that the configured shape must match the format.
-	ori_shape	This function is optional. Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the originally defined one.
-	typical_shape	This function is optional. Int, 1D or 2D array. Shape for test. If the shape field contains a -1 placeholder, add the typical_shape field in the operator test case definition file to specify a static shape value for test. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	shape_range	This function is optional. Int. 1D or 2D array. Value range of shape when the operator supports dynamic shape. Defaults to [[1, -1]], indicating that shape ranges from 1 to infinity. For example, if shape is set to (200, -1) and shape_range is set to [[1, -1]], the value of the second dimension of shape ranges from 1 to infinity. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	is_const	This function is optional. Bool. true: configures test cases with constant inputs. false: configures test cases with tensor inputs. Click here to see the configuration example of an operator with constant input.
-	data_distribute	Required. A string or a 1D array of strings. Data distribution modes for generating test data. uniform: returns random values that are evenly distributed. normal: returns random values of the normal distribution (Gaussian distribution). beta: returns random values of Beta distribution. laplace: returns random values of Laplace distribution. triangular: returns random values of the triangular distribution. relu: returns random values that are evenly distributed and activated by the ReLU function. sigmoid: returns random values that are evenly distributed and activated by the sigmoid function. softmax: returns random values that are evenly distributed and activated by the softmax function. tanh: returns random values that are evenly distributed and activated by the tanh function. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	value_range	Required. Int or float. 1D or 2D array. Value range. Must not be left empty. Formatted as [min_value, max_value], with min_value <= max_value. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	value	Optional. String or tensor array. Add this field to specify the input value. Two configuration methods are available: Enter the tensor data (for example, [1,2,3,4]): "value": [1,2,3,4] Specify the path of a binary file (for example, test.bin): "value": "../test.bin" Prepare the .bin file by yourself. The path can be an absolute path or a relative path of the test case definition file. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. NOTE: If the value field is added, the data_distribute and value_range fields will be ignored. In addition, ensure that the value field matches the format, type, and shape fields. Each test case supports one data type only. Click here to see the configuration example.
output_desc	-	(Required) Operator output description. NOTICE: The number of output_desc's fields must be the same as that of input_desc's fields. Failure to do so may result in test case generation failures. For example, if inputx supports two formats, the output should also support two formats.
-	name	This function is optional. A string. Output name. Required for a dynamic multi-output operator. Set it in the format *{OutputName}{ID}* based on outputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four outputs, there must be corresponding configuration for each output in output_desc and their names are xxx0, xxx1, xxx2, and xxx3, where *xxx* indicates the name of an output.
-	format	Required. A string or a 1D array of strings. Output layout. Must not be empty. Selected from: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 × W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 × W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 x W0) indicates the size of a fractal, as shown in the following figure. fuzz: automatically generates test cases in batches by using the fuzzing script.
-	ori_format	This function is optional. A string or a 1D array of strings. This field is required if the original format is not consistent with the original defined one. In this case, set it to the original format. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented format is the original defined one.
-	type	Required. String, 1D array of strings, or "fuzz". Supported output data types: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float bfloat16 (supported only by Atlas A3 training products/Atlas A3 inference products and Atlas A2 Training Series Product/Atlas 800I A2 Inference Product) fuzz: automatically generates test cases in batches by using the fuzzing script.
-	shape	Required. Int: 1D or 2D array. Input shape. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
-	ori_shape	Optional. Int, 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the original defined one.
attr	-	This function is optional.
-	name	Required if attr is configured. A string. Attribute name. Must not be empty.
-	type	Required if attr is configured. A string. Supported attribute types: bool int float string list_bool list_int list_float list_string list_list_int data_type: Set type to data_type if value of the attr field is a data type.
-	value	Required if attr is configured. Attribute value, which is type-specific. If type is set to bool, value must be true or false. If type is set to int, value must be an int. If type is set to float, value must be a floating point number. If type is set to string, value must be a string, for example, NCHW. If type is set to list_bool, value must be a list of bools, for example, [false, true]. If type is set to list_int, value must be a list of ints, for example, [1, 224, 224, 3]. If type is set to list_float, value must be a list of floats, for example, [1.0, 0.0]. If type is set to list_string, value must be a list of strings, for example, ["str1", "str2"]. If type is set to list_list_int, value must be a list of int lists, for example, [[1, 3, 5, 7], [2, 4, 6, 8]]. If type is set to data_type, value can be int8, int32, int16, int64, uint8, uint16, uint32, uint64, float, float16, float32, bool, double, complex64, complex128, or bfloat16. When value is set to fuzz, values of this parameter are automatically generated as part of the test cases generated in batches using the fuzzing script.
calc_expect_func_file	-	This function is optional. String File path including the name of the function for generating expected operator data, for example, *"/home/test/test_.py:function". Replace /home/test/test_.py* with the implementation file of the expected operator data generation function, and *function* with the function name. NOTICE: You are advised not to call the script for generating expected data in the directory of another user to avoid privilege escalation risks.

(Optional) Customize a data generation function if you want to compare the expected operator outputs with actual outputs. For details, refer to the following steps.
1. Customize a function for generating the expected data of operator Add.
  Implement the function in Python. The file directory including the file name can be user-defined, for example, /home/test/test_add_st.py.
  
  For example, the expected operator data generation function of the Add operator is implemented as follows:
  1 2 3
  def calc_expect_func(x1, x2, y): res = x1["value"] + x2["value"] return [res, ]
  You need to create the expected data generation function of the operator based on the developed custom operator. The name of all input, output, and attribute elements in the test case definition file are used as the input parameters of the expected data generation function of the operator. If an input is optional, the default value will be specified for the input.
  
  For example, if the x3 input is optional, define the expected operator data generation function of the operator as follows:
  1
  def calc_expect_func(x1, x2, x3=None, y=None)
2. Add a comparison function to the ST case definition file, that is, OpType_xx.json. Edit the test case definition file as required.
  Add the calc_expect_func_file parameter to the operator test case definition file AddCustom_case_timestamp.json in step 2. The parameter value is /home/test/test_add_st.py:calc_expect_func.
```
[
    {
        "case_name":"Test_AddCustom_001",         
        "op": "AddCustom",                             
        "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func",   // Configure the implementation file for generating expected compute result.
        "input_desc": [...]
        ...
        ...
    }
]
```

Parent topic: msOpST (Operator Test)