Generating a Case Definition File

Procedure

1. Obtain the path of the information library definition file (.ini) of the operator to be tested.

   The msopst tool generates an ST case definition file for your operator based on the operator information library definition file (.ini). Find the operator information library definition file in the operator project directory as follows:

   ├── cpukernel                             // AI CPU operator file directory
   │   ├── CMakeLists.txt
   │   ├── impl        // Directory of operator code implementation files
   │   │   ├── xx_kernels.cc
   │   │   └── xx_kernels.h
   │   ├── op_info_cfg
   │   │   ├── xx_kernel
   │   │       ├── xx.ini                // Operator information library definition file
   │   └── toolchain.cmake
   ├── tbe                                    // Directory of TBE operator files
   │   ├── CMakeLists.txt   
   │   ├── impl       // Directory of operator implementation files
   │       └── xx.py       // Operator implementation file
   │   ├── op_info_cfg     // Directory of the operator information library files
   │       └── ai_core
   │                 ├── {Soc Version}        // Ascend AI Processor version
   │                     ├── xx.ini         // Operator information library definition file

   The .ini file is stored in tbe/op_info_cfg/ai_core/{Soc Version}/xx.ini in the TBE operator project directory. {Soc Version} indicates the version of Ascend AI Processor in use.

   The .ini file is stored in cpukernel/op_info_cfg/aicpu_kernel/xx.ini of your AI CPU operator project.

   

   If you are performing ST on an AI CPU custom operator, do not change the directory structure of the operator project. The tool will find the operator prototype definition file in the operator project directory where the operator information library definition file is located, and generate the operator test case definition file based on the operator prototype.

2. Go to the directory where the msopst tool is located, and run the following command to generate an operator test case definition file. Table 1 describes the command-line options.

   ./msopst create -i {operator define file} -out {output path} -m {pb file} -q

   Table 1 Command-line options

   Option	Description	Required
   create	Generates an operator test case definition file (.json).	Yes
   -i, --input	Sets the path of the operator information library definition file (.ini). The path can be either absolute or relative. Note: The input operator information library definition file (.ini) can contain the definition of one operator only.	Yes
   -out, --output	Sets the output path. The path can be either absolute or relative. The user who runs the tool must have the read and write permissions on the path. If this option is not specified, the outputs are generated to the current path where the command is executed.	No
   -m, --model	Sets the path of the TensorFlow model file. The path can be either absolute or relative. If this option is set, the tool obtains the shape of the input layer from the TensorFlow model file and automatically dumps the shape, dtype, and attribute values of the operator in the operator information library definition file. If the dumped results are within the valid ranges configured in the operator information library definition file, they are automatically copied to the generated operator test case definition file. Otherwise, an error is reported. NOTICE: To use this option, TensorFlow 1.15 or 2.6.5 is required.	No
   -q, --quiet	Enables quiet generation, which applies only to the -m option. If the -q option is not included, the system prompts you to modify the input layer shape in the obtained model. If the -q option is included, the system does not prompt you to modify the input layer shape.	No

   Example:

   Take the Add operator as an example. Go to the directory where the msopst tool is located and run the following command:

   ./msopst create -i OpInfoDefine/add.ini -out ./output

   Set OpInfoDefine to the directory of the operator information library definition file. For details, see Step 1.

   After the command is executed successfully, an operator test case definition file Add_case_timestamp.json is generated to the output directory of the current path.

3. Modify the test case definition template file OpType_case_timestamp.json.
   Edit the .json file as follows to meet the ST coverage requirements. This step is required because the operator test case definition file (.json) generated in Step 2 is a template file and cannot be used directly for generating ST cases.

   [
         {
           "case_name":"Test_Add_001",
           "error_threshold":[0.1,0.1],
           "st_mode":"pt_python_train", 
           "run_torch_api":"torch.square",
           "op": "Add",
           "input_desc": [ 
               { 
                   "name": "x1",
                   "format": [
                       "ND"
                   ],
                   "type": [ 
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16], 
                   "data_distribute": [
                       "uniform"                 
                   ],
                   "value_range": [
                       [
                           0.1,
                           1.0
                       ]
                   ]
               },
               { 
                   "name": "x2",
                   "format": [
                       "ND"
                   ],
                    "type": [
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16], 
                   "data_distribute": [
                       "uniform"
                   ],
                   "value_range": [
                       [
                           0.1,
                           1.0
                       ]
                   ]
               }
           ],  
           "output_desc": [
               {
                   "name": "y",
                   "format": [
                       "ND"
                   ],
                   "type": [
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16] 
             }
           ]
       }
   ]

   The rest configuration options and their description of the test case definition file is as follows:

   • For inputx, when inputx.paramType in the TBE operator information library definition file (.ini) is set to optional, format is UNDEFINED or RESERVED and type is UNDEFINED in the generated operator test cases.
   • For inputx, when inputx.paramType in the TBE operator information library definition file (.ini) is set to dynamic, name is in the format {OpName}{ID}. ID is indexed based on the number of dynamic inputs and starts from 0 in ascending order.
   • In the TBE operator information library definition file (.ini), if a tensor uses an implementation format different from its original format, manually add the ori_format and ori_shape fields to convert origin_format and origin_shape into the format and shape compatible with the offline model.
     • ori_format is an array specifying the formats supported by the original operator. The array must have the same length as the format array.
     • ori_shape is the converted shape based on format and ori_format.

   The template requires manual tweaks. The following table describes all fields to be configured in the .json file. For examples of test case definition files in different scenarios, see Configuration Example of the Test Case Definition File.

   Table 2 Operator test case definition file (.json)

   Field		Description
   case_name	-	(Required) String. Test case name.
   op	-	(Required) String. Operator type. Must not be empty.
   error_threshold	-	(Optional) Customizes the precision standard. The value is a list containing two elements, formatted as "[threshold1,threshold2]". threshold1: threshold of the error between the operator output result and the benchmark data. If the actual error is greater than this threshold, the operator output result is recorded as the error data. threshold2: threshold of the ratio of error data to all data. If the actual ratio is less than this threshold, the precision is deemed acceptable. Otherwise, the precision does not meet the requirement. Element value range: [0.0, 1.0]. NOTE: If this parameter is set both in the JSON file of the test case and in the corresponding msopst command, the latter value is used for comparison. If this parameter is not set in neither of them, the default precision standard [0.01, 0.05] set in the msopst command is used for comparison.
   st_mode	-	(Optional) String. ST mode. The value ms_python_train indicates a MindSpore operator project (only supported by the Atlas Training Series Product), and pt_python_train indicates a PyTorch operator project. Atlas 200/300/500 Inference Product does not support this parameter.
   run_torch_api	-	(Optional) Sets the API for torch_api to call the operator. The value is torch.square, in which square indicates the API name. Set this parameter as required. Atlas 200/300/500 Inference Product does not support this parameter.
   expect	-	(Optional) Expected test result. It can be either of the following values. The default value is success. success: execution success. If the model conversion fails, the process will be terminated. You can view ATC logs to locate the fault. failed: execution failure. If you need to run an abnormal case, change the value of the expect field to failed. If the model conversion fails, the process continues. If the statistical results of status and expect in the ST case report are consistent, the test cases are included in "success count." Otherwise, the test cases are included in "failed count."
   fuzz_impl	-	(Optional) String. Required if you need plenty of test cases that can be generated by using a fuzzing script. In this case, add this field and set it to "relative or absolute path of the fuzzing script:function name". For details about how to implement this script, see 4. NOTE: You are advised not to invoke the fuzzing script in the directory of another user to avoid privilege escalation risks.
   fuzz_case_num	-	(Optional) Int. Number of test cases generated using the fuzzing script. Required if the fuzz_impl field is added. The value ranges from 1 to 2000.
   input_desc	-	(Required) Operator input description. NOTICE: The parameters in input_desc must have the same number of values. Failure to do so may result in test case generation failures. For example, if input1 supports two formats, input2 should also support two formats. The rule also applies to type, shape, data_distribute, and value_range of all inputs.
   -	name	(Optional) Required for a dynamic multi-input operator. Set it in the format {InputName}{ID} based on inputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four inputs, there must be corresponding configuration for each input in input_desc and their names are xxx0, xxx1, xxx2, and xxx3, where xxx indicates the name of an input. Click here to see the configuration example of an operator with uncertain number of inputs (dynamic multi-input operator).
   -	format	(Required) String or 1D array of strings. Input format. Must not be empty. Common data formats are as follows: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. The matrix is divided into (H1 x W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 x W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 x W0) indicates the size of a fractal, as shown in the following figure. RESERVED: reserved. If this value is used, type must be UNDEFINED, indicating that the corresponding operator input is optional. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	ori_format	(Optional) String or 1D array of strings. Set it in either of the following ways: Set it to the original format of the input. This field is required if the original operator format is not consistent with the implemented one. If this field is not configured, the original format is preserved. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	type	(Required) String or 1D array of strings. Input data type: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float double (supported only by AI CPU operators) complex64 (supported only by AI CPU operators) complex128 (supported only by AI CPU operators) UNDEFINED: applies to an optional operator input. fuzz: automatically generates test cases in batches by using the fuzzing script. Click here to see the configuration example of an operator with the input of a complex number.
   -	shape	(Required) Int. 1D or 2D array. Input shape. For static shape: The dimensions and value of shape are fixed and shape_range does not need to be configured. For dynamic shape: If the shape values contain a -1 placeholder, for example, (200, -1), the size of the second dimension is unknown. In this scenario, this parameter must be used together with shape_range to specify the value range that -1 supports. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. Left empty. Can be empty if format and type are set to UNDEFINED. Note that the configured shape must match the format.
   -	ori_shape	(Optional) Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the original defined one.
   -	typical_shape	(Optional) Int. 1D or 2D array. Shape for test. If the shape field contains a -1 placeholder, add the typical_shape field in the operator test case definition file to specify a static shape value for test. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	shape_range	(Optional) Int. 1D or 2D array. Value range of shape when the operator supports dynamic shape. Defaults to [[1,-1]], indicating that shape ranges from 1 to infinity. For example, if shape is set to (200, -1) and shape_range is set to [[1, -1]], the value of the second dimension of shape ranges from 1 to infinity. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	is_const	(Optional) Bool. true: configures test cases with constant inputs. false: configures test cases with tensor inputs. Click here to see the configuration example of an operator with constant input.
   -	data_distribute	(Required) String or 1D array of strings. Data distribution modes for generating test data. uniform: returns random values that are evenly distributed. normal: returns random values of the normal distribution (Gaussian distribution). beta: returns random values of Beta distribution. laplace: returns random values of Laplace distribution. triangular: returns random values of the triangular distribution. relu: returns random values that are evenly distributed and activated by the ReLU function. sigmoid: returns random values that are evenly distributed and activated by the sigmoid function. softmax: returns random values that are evenly distributed and activated by the softmax function. tanh: returns random values that are evenly distributed and activated by the tanh function. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	value_range	(Required) Int or float: 1D or 2D array. Value range. Must not be empty. Formatted as [min_value, max_value], with min_value <= max_value. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	value	(Optional) String or tensor. Add this field to specify the input value. Two configuration methods are available: Enter the tensor data (for example, [1,2,3,4]): "value": [1,2,3,4] Specify the path of a binary file (for example, test.bin): "value": "../test.bin" Prepare the .bin file by yourself. The path can be an absolute path or a relative path of the test case definition file. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. NOTE: If the value field is added, the data_distribute and value_range fields will be ignored. In addition, ensure that the value field matches the format, type, and shape fields. Each test case supports one data type only. Click here to see the configuration example of an operator with fixed input.
   output_desc	-	(Required) Operator output description. NOTICE: The number of output_desc's fields must be the same as that of input_desc's fields. Failure to do so may result in test case generation failures. For example, if inputx supports two formats, the output should also support two formats.
   -	name	(Optional) String. Output name. Required for a dynamic multi-output operator. Set it in the format {OutputName}{ID} based on outputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four outputs, there must be corresponding configuration for each output in output_desc and their names are xxx0, xxx1, xxx2, and xxx3, where xxx indicates the name of an output.
   -	format	(Required) String or 1D array of strings. Output format. Must not be empty. Selected from: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. The matrix is divided into (H1 x W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 x W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 x W0) indicates the size of a fractal, as shown in the following figure. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	ori_format	(Optional) String or 1D array of strings. This field is required if the original format is not consistent with the original defined one. In this case, set it to the original format. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented format is the original defined one.
   -	type	(Required) String, 1D array of strings, or "fuzz". Output data type. bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float double (supported only by AI CPU operators) complex64 (supported only by AI CPU operators) complex128 (supported only by AI CPU operators) fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	shape	(Required) Int. 1D or 2D array. Input shape. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	ori_shape	(Optional) Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: "fuzz". Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the original defined one.
   attr	-	(Optional)
   -	name	Required if attr is configured. String. Attribute name. Must not be empty.
   -	type	Required if attr is configured. String. Attribute type: bool int float string list_bool list_int list_float list_string list_list_int data_type: Set type to data_type if value of the attr field is a data type.
   -	value	Required if attr is configured. Attribute value, which is type-specific. If type is set to bool, value must be true or false. If type is set to int, value must be an int. If type is set to float, value must be a floating point number. If type is set to string, value must be a string, for example, NCHW. If type is set to list_bool, value must be a list of bools, for example, [false, true]. If type is set to list_int, value must be a list of ints, for example, [1, 224, 224, 3]. If type is set to list_float, value must be a list of floats, for example, [1.0, 0.0]. If type is set to list_string, value must be a list of strings, for example, ["str1", "str2"]. If type is set to list_list_int, value must be a list of int lists, for example, [[1, 3, 5, 7], [2, 4, 6, 8]]. If type is set to data_type, value can be int8, int32, int16, int64, uint8, uint16, uint32, uint64, float, float16, float32, bool, double, complex64, complex128, or bfloat16. When value is set to fuzz, values of this parameter are automatically generated as part of the test cases generated in batches using the fuzzing script.
   calc_expect_func_file	-	(Optional) String. File path including the name of the function for generating expected operator data, for example, "/home/teste/test_*.py:function". Replace /home/teste/test_*.py with the implementation file of the expected operator data generation function, and function with the function name. NOTICE: You are advised not to invoke the script for generating expected data in the directory of another user to avoid privilege escalation risks.

4. Implement a fuzzing script (.py) and configure a test case definition file (.json) as needed to automatically generate a wealth of test cases. The procedure is as follows:
   1. Implement the fuzzing script, which automatically generates the parameters other than name in the input_desc, output_desc, and attr fields in the test case definition file.
      The following demonstrates how to generate random shape and value arguments through an example script (fuzz_shape.py). In this example, the shape for ST is dynamic. The dimension size ranges from 1 to 4, and the value of each dimension ranges from 1 to 64.

      1. Import the required dependency.

         import numpy as np

      2. Implement the fuzz_branch() method. If you customize the name of the method for randomly generating the parameters to be tested, you need to set the fuzz_impl field in the operator test case definition file.

         def fuzz_branch():
             # Generate shape values for testing.
             dim = random.randint(1, 4)
             x_shape_0 = random.randint(1, 64)
             x_shape_1 = random.randint(1, 64)
             x_shape_2 = random.randint(1, 64)
             x_shape_3 = random.randint(1, 64)
             if dim == 1:
                 shape = [x_shape_0]
             if dim == 2:
                 shape = [x_shape_0, x_shape_1]
             if dim == 3:
                 shape = [x_shape_0, x_shape_1, x_shape_2]
             if dim == 4:
                 shape = [x_shape_0, x_shape_1, x_shape_2, x_shape_3]
             # Randomly generate x1 and x2's values according to the shape argument.
             fuzz_value_x1 = np.random.randint(1, 10, size=shape)
             fuzz_value_x2 = np.random.randint(1, 10, size=shape)
         
             # Return shape values of type dictionary to input_desc's x1 and x2 and output_desc's y. In the test case definition file, x1 and x2 are the inputs' names and y is the output's name.
             return {"input_desc": {"x1": {"shape": shape,"value": fuzz_value_x1}, 
                                    "x2": {"shape": shape,"value": fuzz_value_x2}},
                     "output_desc": {"y": {"shape": shape}}}

         • This method is used to generate arguments except the name arguments of input_desc, output_desc, and attr in the test case definition file. You can also customize the generation method for operator test as needed.
         • The returns are of the dictionary type and assigned to the operator in this type for ST. The returned dictionary is the same as the parameter structure in the test case definition file.
   2. Configure the test case definition file.
      1. Add the fuzz_impl field. The value is in the format "relative or absolute path of the fuzzing script:function name", for example, "conv2d_fuzz.py:fuzz_branch". If you want to use your custom method for randomly generating parameters, replace fuzz_branch with the method name. If the function name is not set, the fuzz_branch method is used.
      2. Add the fuzz_case_num field with the value set to the number of test cases generated by the fuzzing script, for example, 2000.
      3. Set the parameters to be automatically generated to fuzz.

         See Table 2 for details about the parameters.

      The following gives an example of the operator test case definition file.

      [
          {
              "case_name":"Test_Add_001",         
              "op": "Add",                             
              "fuzz_impl": "./fuzz_shape.py:fuzz_branch",   // Set the value to "directory of the fuzzing script:function name"
              "fuzz_case_num": 2000,   // Configure the number of test cases to be generated.
               "input_desc": [            // Input description
                  {                     // The first input
                      "name": "x1",
                      "format": [
                          "ND"       // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"  // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape":"fuzz",     // fuzz indicates the value will be automatically generated.
                      "data_distribute": [
                          "uniform"
                      ],
                      "value_range": [
                          [
                              0.1,
                              1.0
                          ]
                      ],	
                      "value": "fuzz"	
                   {                     // The second input
                      "name": "x2",
                       "format": [
                          "ND"          // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"     // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape": "fuzz",  // fuzz indicates the value will be automatically generated.
                      "data_distribute": [
                          "uniform"
                      ],
                      "value_range": [
                          [
                              0.1,
                              1.0
                          ]
                      ],       
                      "value": "fuzz" 
                  }
              ],  
              "output_desc": [                       // (Required) Output description. The detailed description is the same as that for input_desc.
                  {
                      "name": "y",
                      "format": [
                          "ND"                     // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"         // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape":"fuzz"       //  fuzz indicates the value will be automatically generated.
          }
      ]

      

      • For each "fuzz" value of an input, output, or attribute in the test case definition file, there must be corresponding "name" parameters. If there are no such "name" parameters, add them manually, for example, "name": "y".
      • If one or more parameter values in the test case definition file are "fuzz", the values of the rest parameters must be unique and compatible with the parameter values generated by the fuzzing script. For example, if the shape is set to "fuzz" and the generated shape is [1, 3, 16, 16], the specified format must be a 4D one.
5. Optionally customize an expected operator data generation function if you want to compare the expected operator output with actual output.
   1. Customize a function for generating the expected data of operator Add.

      Implement the function in Python. The file directory including the file name can be user-defined, for example, /home/teste/test_add_st.py.

      For example, the expected operator data generation function of the Add operator is implemented as follows.

      def calc_expect_func(x1, x2, y):    
          res = x1["value"] + x2["value"]
          return [res, ]

      

      You need to create the expected data generation function of the operator based on the developed custom operator. The name of all input, output, and attribute elements in the test case definition file are used as the input parameters of the expected data generation function of the operator. If an input is optional, the default value will be specified for the input.

      For example, if the x3 input is optional, define the expected operator data generation function of the operator as follows:

      def calc_expect_func(x1, x2, x3=None, y=None)

   2. Add a comparison function to the ST case definition file, that is, OpType_xx.json. Edit the test case definition file as required.
      For example, in the operator test case definition file Add_case_timestamp.json generated in 1, add the "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func" line.

      [
          {
              "case_name":"Test_Add_001",         
              "op": "Add",                             
              "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func",   // Configure the implementation file for generating expected compute result.
              "input_desc": [...]
              ...
              ...
          }
      ]