Generating a Case Definition File

Procedure

1. Obtain the path of the information library definition file (.ini) of the operator to be tested.

   The msOpST tool generates an ST case definition file for your operator based on the operator information library definition file (.ini). Find the operator information library definition file in the operator project directory as follows:

   ├── cpukernel                             // AI CPU operator file directory
   │   ├── CMakeLists.txt
   │   ├── impl        // Directory of operator code implementation files
   │   │   ├── xx_kernels.cc
   │   │   └── xx_kernels.h
   │   ├── op_info_cfg
   │   │   ├── xx_kernel
   │   │       ├── xx.ini                // Operator information library definition file
   │   └── toolchain.cmake
   ├── tbe                                    // Directory of TBE operator files
   │   ├── CMakeLists.txt   
   │   ├── impl       // Directory of operator implementation files
   │       └── xx.py      // Operator code implementation file
   │   ├── op_info_cfg     // Directory of the operator information library files
   │       └── ai_core
   │                 ├── {Soc Version}        // Ascend AI Processor version
   │                     ├── xx.ini         // Operator information library definition file

   The .ini file is stored in tbe/op_info_cfg/ai_core/{Soc Version}/xx.ini in the TBE operator project directory. {Soc Version} indicates the version of Ascend AI Processor.

   The .ini file is stored in cpukernel/op_info_cfg/aicpu_kernel/xx.ini of your AI CPU operator project.

   

   If you are performing ST on an AI CPU custom operator, do not change the directory structure of the operator project. The tool will find the operator prototype definition file in the operator project directory where the operator information library definition file is located, and generate the operator test case definition file based on the operator prototype.

2. Run the following command to generate an operator test case definition file. Table 1 describes the available command-line arguments.

   msopst create -i {operator define file} -out {output path} -m {pb file} -q

   Table 1 Parameters

   Parameter	Description	Mandatory (Yes/No)
   create	Generates an operator test case definition file (.json).	Yes
   -i, --input	Sets the path of the operator information library definition file (.ini). The path can be either absolute or relative. Note: The input operator information library definition file (.ini) can contain the definition of one operator only.	Yes
   -out, --output	Sets the output path. The path can be either absolute or relative. The user who runs the tool must have the read and write permissions on the path. If this option is not specified, the outputs are generated to the current path where the command is executed.	No
   -m, --model	Sets the path of the TensorFlow model file. The path can be either absolute or relative. If this option is set, the tool obtains the shape of the input layer from the TensorFlow model file and automatically dumps the shape, dtype, and attribute values of the operator in the operator information library definition file. If the dumped results are within the valid ranges configured in the operator information library definition file, they are automatically copied to the generated operator test case definition file. Otherwise, an error is reported. NOTE: To use this option, TensorFlow 1.15 or 2.6.5 is required.	No
   -q, --quiet	Enables quiet generation, which applies only to the -m option. If the -q option is not included, the system prompts you to modify the top-layer shape in the obtained model. If the -q option is included, the system does not prompt you to modify the top-layer shape.	No
   -h, --help	Outputs the help information.	No

   Example:

   Take the add operator as an example. Run the following command:

   msopst create -i OpInfoDefine/add.ini -out ./output

   Set OpInfoDefine to the directory of the operator information library definition file. For details, see 1.

   After the command is executed successfully, an operator test case definition file Add_case_timestamp.json is generated to the output directory of the current path.

3. Modify the test case definition template file OpType_case_timestamp.json.
   Edit the .json file as follows to meet the ST coverage requirements. This step is required because the operator test case definition file (.json) generated in 2 is a template file and cannot be used directly for generating ST cases.

   [
         {
           "case_name":"Test_Add_001",
           "error_threshold":[0.1,0.1],
           "st_mode":"pt_python_train", 
           "run_torch_api":"torch.square",
           "op": "Add",
           "input_desc": [ 
               { 
                   "name": "x1",
                   "format": [
                       "ND"
                   ],
                   "type": [ 
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16], 
                   "data_distribute": [
                       "uniform"                 
                   ],
                   "value_range": [
                       [
                           0.1,
                           1.0
                       ]
                   ]
               },
               { 
                   "name": "x2",
                   "format": [
                       "ND"
                   ],
                    "type": [
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16], 
                   "data_distribute": [
                       "uniform"
                   ],
                   "value_range": [
                       [
                           0.1,
                           1.0
                       ]
                   ]
               }
           ],  
           "output_desc": [
               {
                   "name": "y",
                   "format": [
                       "ND"
                   ],
                   "type": [
                       "int32",
                       "float",
                       "float16"
                   ],
                   "shape": [32,16] 
             }
           ]
       }
   ]

   The rest configuration options and their description of the test case definition file is as follows:

   • For inputx, when inputx.paramType in the TBE operator information library definition file (.ini) is set to optional, format is UNDEFINED or RESERVED and type is UNDEFINED in the generated operator test cases.
   • For inputx, when inputx.paramType in the TBE operator information library definition file (.ini) is set to dynamic, name is in the format {OpName}{ID}. ID is indexed based on the number of dynamic inputs and starts from 0 in ascending order.
   • In the TBE operator information library definition file (.ini), if a Tensor uses an implementation format different from its original format, manually add the ori_format and ori_shape fields to convert origin_format and origin_shape into the format and shape compatible with the offline model.
     • ori_format is an array specifying the formats supported by the original operator. The array must have the same length as the format array.
     • ori_shape is the converted shape based on format and ori_format.

   The template requires manual tweaks. The following table describes all fields to be configured in the .json file. For examples of test case definition files in different scenarios, see Configuration Example of the Test Case Definition File.

   Table 2 Operator test case definition file (.json)

   Parameter		Description
   case_name	-	Mandatory. String. Test case name.
   op	-	Mandatory. String. Operator type. The value cannot be empty.
   error_threshold	-	Optional. Customizes the accuracy standard. The value is a list containing two elements, formatted as [threshold1, threshold2]. threshold1: threshold of the error between the operator output result and the benchmark data. If the actual error is greater than this threshold, the operator output result is recorded as the error data. threshold2: threshold of the ratio of error data to all data. If the actual ratio is less than this threshold, the accuracy is deemed acceptable. Otherwise, the precision does not meet the requirement. Element value range: [0.0, 1.0]. NOTE: If this parameter is set both in the JSON file of the test case and in the corresponding msopst command, the latter value is used for comparison. If this parameter is not set in neither of them, the default precision standard [0.01, 0.05] set in the msopst command is used for comparison.
   st_mode	-	Optional. String. ST mode. The value ms_python_train indicates a MindSpore operator project (only supported by the Atlas training products), and pt_python_train indicates a PyTorch operator project.
   run_torch_api	-	Optional. Sets the API for torch_api to call the operator. The value is torch.square, in which square indicates the API name. Set this parameter as needed.
   expect	-	Optional. Expected test result. It can be either of the following values. The default value is success. success: execution success. If the model conversion fails, the process will be terminated. You can view ATC logs to locate the fault. failed: execution failure. If you need to run an abnormal case, change the value of the expect field to failed. If the model conversion fails, the process continues. If the statistical results of status and expect in the ST case report are consistent, the test cases are included in "success count." Otherwise, the test cases are included in "failed count."
   fuzz_impl	-	(Optional) String. Required if you need to use a fuzzing script to generate plenty of test cases. In this case, add this field and set it to relative or absolute path of the fuzzing script:function name. For details about how to implement this script, see 4. NOTE: You are advised not to call the fuzzing script in the directory of another user to avoid privilege escalation risks.
   fuzz_case_num	-	Optional. Int. Number of test cases generated using the fuzzing script. Required if the fuzz_impl field is added. The value ranges from 1 to 2000.
   input_desc	-	Mandatory. Operator input description. NOTE: The parameters in input_desc must have the same number of values. Failure to do so may result in test case generation failures. For example, if input1 supports two formats, input2 should also support two formats. The rule also applies to type, shape, data_distribute, and value_range of all inputs.
   -	name	Optional. Required for a dynamic multi-input operator. Set it in the format {InputName}{ID} based on inputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four inputs, there must be corresponding configuration for each input in input_desc and their names are xxx0, xxx1, xxx2, and xxx3, where xxx indicates the name of an input. Click atlasopdev_10_0099.html#EN-US_TOPIC_0000002502586554__li847855219617 to see the configuration example of an operator with uncertain number of inputs (dynamic multi-input operator).
   -	format	Mandatory. String or 1D array of strings. Format of the input tensor. Must not be empty. Common data formats are as follows: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 × W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 × W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 × W0) indicates the size of a fractal, as shown in the following figure. RESERVED: reserved. If this value is used, type must be UNDEFINED, indicating that the corresponding operator input is optional. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	ori_format	Optional. String or 1D array of strings. Set it in either of the following ways: Set it to the original format of the input. This field is required if the original operator format is not consistent with the implemented one. If this field is not configured, the original format is preserved. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script.
   -	type	Mandatory. String or 1D array of strings. Input data type: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float bfloat16 (supported only by Atlas A2 training products/Atlas A2 inference products) double (supported only by AI CPU operators) complex64 (supported only by AI CPU operators) complex128 (supported only by AI CPU operators) UNDEFINED: applies to an optional operator input. fuzz: automatically generates test cases in batches by using the fuzzing script. Click atlasopdev_10_0099.html#EN-US_TOPIC_0000002502586554__li115917150720 to see the configuration example of an operator with the input of a complex number.
   -	shape	Mandatory. Int. 1D or 2D array. Input tensor shape. For static shape: The dimensions and value of shape are fixed and shape_range does not need to be configured. For dynamic shape: If the shape values contain a -1 placeholder, for example, (200, -1), the size of the second dimension is unknown. In this scenario, this parameter must be used together with shape_range to specify the value range that -1 supports. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. Left empty. If format and type are set to UNDEFINED, the shape parameter can be empty. Note that the configured shape must match the format.
   -	ori_shape	Optional. Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the originally defined one.
   -	typical_shape	Optional. Int. 1D or 2D array. Shape for test. If the shape field contains a -1 placeholder, add the typical_shape field in the operator test case definition file to specify a static shape value for test. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
   -	shape_range	Optional. Int. 1D or 2D array. Value range of shape when the operator supports dynamic shape. The default value is [[1,-1]]. It indicates that shape ranges from 1 to infinity. For example, if shape is set to (200, -1) and shape_range is set to [[1, -1]], the value of the second dimension of shape ranges from 1 to infinity. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
   -	is_const	Optional. Bool. true: configures test cases with constant inputs. false: configures test cases with tensor inputs. Click atlasopdev_10_0099.html#EN-US_TOPIC_0000002502586554__li1664717410582 to see the configuration example of an operator with constant input.
   -	data_distribute	Mandatory. String or 1D array of strings. Data distribution modes for generating test data. uniform: returns random values that are evenly distributed. normal: returns random values of the normal distribution (Gaussian distribution). beta: returns random values of Beta distribution. laplace: returns random values of Laplace distribution. triangular: returns random values of the triangular distribution. relu: returns random values that are evenly distributed and activated by the ReLU function. sigmoid: returns random values that are evenly distributed and activated by the sigmoid function. softmax: returns random values that are evenly distributed and activated by the softmax function. tanh: returns random values that are evenly distributed and activated by tanh function. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	value_range	Mandatory. Int or float. 1D or 2D array. Value range. Must not be left empty. Formatted as [min_value, max_value], with min_value ≤ max_value. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
   -	value	Optional. String or tensor. If you need to specify the input data, you can add the value field. Two configuration methods are available: Enter the tensor data (for example, [1,2,3,4]): "value": [1,2,3,4] Specify the path of a binary file (for example, test.bin): "value": "../test.bin" Prepare the .bin file by yourself. The path can be an absolute path or a relative path of the test case definition file. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. NOTE: If the "value" field is added, the "data_distribute" and "value_range" fields will be ignored. In addition, ensure that the "value" field matches the "format", "type", and "shape" fields. Each test case supports one data type only. Click atlasopdev_10_0099.html#EN-US_TOPIC_0000002502586554__li10983136112016 to see the configuration example of an operator with fixed input.
   output_desc	-	Mandatory. Operator output description. NOTE: The number of parameter values in output_desc must be the same as that in input_desc. Failure to do so will result in test case generation failures. For example, if inputx supports two formats, the output should also support two formats.
   -	name	Optional. String. Output name. Required for a dynamic multi-output operator. Set it in the format {OutputName}{ID} based on outputx.name in the operator information library. ID starts from 0 in ascending order. For example, if the operator information library definition file specifies four outputs, there must be corresponding configuration for each output in output_desc and their names are xxx0, xxx1, xxx2, and xxx3, where xxx indicates the name of an output.
   -	format	Mandatory. String or 1D array of strings. Format of the output tensor. Must not be empty. Selected from: NCHW NHWC ND: any format. NC1HWC0: a 5D format. C0 is closely related to the micro-architecture, and the value is equal to the Cube Unit size, for example, 16. C1 is obtained by dividing the C dimension by C0, that is, C1 = C/C0. When the division is not exact, the last data segment is padded to C0. FRACTAL_Z: a format of the convolution weight. FRACTAL_NZ: a fractal format. The format of the output matrix is NW1H1H0W0 during Cube Unit computation. A matrix is divided into (H1 × W1) fractals in column-major order, which looks like an N-shape layout. Each fractal consists of (H0 × W0) elements in row-major order, resembling a z-shaped layout. Therefore the NW1H1H0W0 format is referred to as the Nz format. (H0 × W0) indicates the size of a fractal, as shown in the following figure. fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	ori_format	Optional. String or 1D array of strings. This field is required if the original format is not consistent with the original defined one. In this case, set it to the original format. Set it to "fuzz" to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented format is the original defined one.
   -	type	Mandatory. String, 1D array of strings, or fuzz. Supported output data types: bool int8 uint8 int16 uint16 int32 int64 uint32 uint64 float16 float32 float bfloat16 (supported only by Atlas A2 training products/Atlas A2 inference products) double (supported only by AI CPU operators) complex64 (supported only by AI CPU operators) complex128 (supported only by AI CPU operators) fuzz: automatically generates test cases in batches by using the fuzzing script.
   -	shape	Mandatory. Int. 1D or 2D array. Input tensor shape. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script.
   -	ori_shape	Optional. Int. 1D or 2D array. Original shape of the input. This field is required if the original operator shape is not consistent with the implemented one. String: fuzz. Set it to fuzz to automatically generate test cases in batches by using the fuzzing script. If this field is not configured, the implemented shape is the originally defined one.
   attr	-	Optional.
   -	name	Required if attr is configured. String. Attribute name. Must not be empty.
   -	type	Required if attr is configured. String. Supported attribute types: bool int float string list_bool list_int list_float list_string list_list_int data_type: Set type to data_type if value of the attr field is a data type.
   -	value	Required if attr is configured. Attribute value, which is type-specific. If type is set to bool, value must be true or false. If type is set to int, value must be an int. If type is set to float, value must be a floating point number. If type is set to string, value must be a string, for example, NCHW. If type is set to list_bool, value must be a list of bools, for example, [false, true]. If type is set to list_int, value must be a list of ints, for example, [1, 224, 224, 3]. If type is set to list_float, value must be a list of floats, for example, [1.0, 0.0]. If type is set to list_string, value must be a list of strings, for example, ["str1", "str2"]. If type is set to list_list_int, value must be a list of int lists, for example, [[1, 3, 5, 7], [2, 4, 6, 8]]. If type is set to data_type, value can be int8, int32, int16, int64, uint8, uint16, uint32, uint64, float, float16, float32, bool, double, complex64, complex128, or bfloat16. When value is set to fuzz, values of this parameter are automatically generated as part of the test cases generated in batches using the fuzzing script.
   calc_expect_func_file	-	Optional. String. File path including the name of the function for generating expected operator data, for example, /home/test/test_*.py:function. Replace /home/test/test_*.py with the implementation file of the expected operator data generation function, and function with the function name. NOTICE: You are advised not to call the script for generating expected data in the directory of another user to avoid privilege escalation risks.

4. Implement a fuzzing script (.py) and configure a test case definition file (.json) as needed to automatically generate a wealth of test cases. The procedure is as follows.
   1. After the fuzzing script is implemented, the parameters other than name of the input_desc, output_desc, and attr fields are automatically generated as part of the test case definition file.
      The following demonstrates how to generate random shape and value parameters through an example script (fuzz_shape.py). In this example, the shape argument for ST is dynamic. The dimension size ranges from 1 to 4, and the value of each dimension ranges from 1 to 64.

      1. Import the dependency required by the script.

         import numpy as np

      2. Implement the fuzz_branch() method. If you customize the name of the method for randomly generating the parameters to be tested, you need to set the fuzz_impl field in the operator test case definition file.

         def fuzz_branch():
             # Generate shape values for testing.
             dim = random.randint(1, 4)
             x_shape_0 = random.randint(1, 64)
             x_shape_1 = random.randint(1, 64)
             x_shape_2 = random.randint(1, 64)
             x_shape_3 = random.randint(1, 64)
             if dim == 1:
                 shape = [x_shape_0]
             if dim == 2:
                 shape = [x_shape_0, x_shape_1]
             if dim == 3:
                 shape = [x_shape_0, x_shape_1, x_shape_2]
             if dim == 4:
                 shape = [x_shape_0, x_shape_1, x_shape_2, x_shape_3]
             # Randomly generate the values of x1 and x2 based on the shape argument.
             fuzz_value_x1 = np.random.randint(1, 10, size=shape)
             fuzz_value_x2 = np.random.randint(1, 10, size=shape)
         
             # Return shape values of type dictionary to input_desc's x1 and x2 and output_desc's y. In the test case definition file, x1 and x2 are the inputs' names and y is the output's name.
             return {"input_desc": {"x1": {"shape": shape,"value": fuzz_value_x1}, 
                                    "x2": {"shape": shape,"value": fuzz_value_x2}},
                     "output_desc": {"y": {"shape": shape}}}

         • This method is used to generate arguments except the name arguments of input_desc, output_desc, and attr in the test case definition file. You can also customize the generation method for operator test as needed.
         • The returns are of the dictionary type and assigned to the operator in this type for ST. The returned dictionary is the same as the parameter structure in the test case definition file.
   2. Configure the test case definition file.
      1. Add the fuzz_impl field. The value is in the format "relative or absolute path of the fuzzing script:function name", for example, "conv2d_fuzz.py:fuzz_branch". If you want to use your custom method for randomly generating parameters, replace fuzz_branch with the method name. If the function name is not set, the fuzz_branch method is used.
      2. Add the "fuzz_case_num" field with the value set to the number of test cases generated by the fuzzing script, for example, 2000.
      3. Set the parameters to be automatically generated to fuzz.

         For details about the parameters, see Table 2.

      The following gives an example of the operator test case definition file.

      [
          {
              "case_name":"Test_Add_001",         
              "op": "Add",                             
              "fuzz_impl": "./fuzz_shape.py:fuzz_branch",   // Set the value to "directory of the fuzzing script:function name"
              "fuzz_case_num": 2000,   // Configure the number of test cases to be generated.
               "input_desc": [            // Input description
                    {                     // The first input
                      "name": "x1",
                      "format": [
                          "ND"       // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"  // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape":"fuzz",     // fuzz indicates the value will be automatically generated.
                      "data_distribute": [
                          "uniform"
                      ],
                      "value_range": [
                          [
                              0.1,
                              1.0
                          ]
                      ],	
                      "value": "fuzz"	
                   {                     // The second input
                      "name": "x2",
                       "format": [
                          "ND"          // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"     // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape": "fuzz",  // fuzz indicates the value will be automatically generated.
                      "data_distribute": [
                          "uniform"
                      ],
                      "value_range": [
                          [
                              0.1,
                              1.0
                          ]
                      ],       
                      "value": "fuzz" 
                  }
              ],  
              "output_desc": [                       // (Mandatory) Output description. The detailed description is the same as that for the input tensor.
                  {
                      "name": "y",
                      "format": [
                          "ND"                     // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "type": [
                          "float16"         // Delete unnecessary values until only one compatible with automatically generated shape values is left.
                      ],
                      "shape":"fuzz"       //  fuzz indicates the value will be automatically generated.
          }
      ]

      

      • For each "fuzz" value of an input, output, or attribute in the test case definition file, there must be corresponding "name" parameters. If there are no such "name" parameters, add them manually, for example, "name": "y".
      • If one or more parameter values in the test case definition file are "fuzz", the values of the rest parameters must be unique and compatible with the parameter values generated by the fuzzing script. For example, if the shape is set to "fuzz" and the generated shape is [1, 3, 16, 16], the specified format must be a 4D one.
5. Optionally customize an expected operator data generation function if you want to compare the expected operator output with actual output.
   1. Customize a function for generating the expected data of the Add operator.

      Implement the function in Python. The file directory including the file name can be user-defined, for example, /home/test/test_add_st.py.

      For example, the expected operator data generation function of the Add operator is implemented as follows:

      def calc_expect_func(x1, x2, y):    
          res = x1["value"] + x2["value"]
          return [res, ]

      

      You need to create the expected data generation function of the operator based on the developed custom operator. The name of all input, output, and attribute elements in the test case definition file are used as the input parameters of the expected data generation function of the operator. If an input is optional, the default value will be specified for the input.

      For example, if the x3 input is optional, define the expected operator data generation function of the operator as follows:

      def calc_expect_func(x1, x2, x3=None, y=None)

   2. Add a comparison function to the ST case definition file, that is, OpType_xx.json. Edit the test case definition file as required.
      For example, in the operator test case definition file Add_case_timestamp.json generated in 1, add the "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func" line.

      [
          {
              "case_name":"Test_Add_001",         
              "op": "Add",                             
              "calc_expect_func_file": "/home/test/test_add_st.py:calc_expect_func",   // Configure the implementation file for generating expected compute result.
              "input_desc": [...]
              ...
              ...
          }
      ]