Overview

After finishing the custom OPP deployment with the msOpGen tool, use the msOpST tool to start the system test (ST) to verify the operator input and output in a real hardware environment.

Test cases usually include various types of data inputs and expected outputs, as well as tests for boundary situations and exceptions. The ST ensures that the operator can function and run properly in actual applications.

In TBE and AI CPU operator development scenarios, for details about how to use the msOpST tool, see Performing Operator ST Tests Using msOpST. The TBE/TIK operator development method is no longer open to external users and the TBE/TIK samples will be brought offline in the next MindStudio version.

Function

msOpST can generate ST cases for operators and execute the cases in the hardware environment. It provides the following functions:

Constructs a user-defined function for generating expected operator output and displays the comparison result between the expected operator output and the actual output. For details, see Generating a Case Definition File.
Generates the ST data and test case execution code based on the operator test case definition file, for running the test cases in the hardware environment. For details, see Generating and Executing Test Cases.
Generates an ST report (st_report.json) that records information about test cases and phase-by-phase execution states. For details, see Generating and Executing Test Cases.
Automatically generate the board test framework for calling the kernel function to test and verify the operator. For details, see Generating Single-Operator Board Test Framework.

Commands

Generate an operator test case definition file.

**Table 1** Parameters for generating an operator test case definition file
Parameter	Description	Required
create	Generates an operator test case definition file (*.json).	Yes
-i, --input	Sets the path of the implementation file (*.cpp) of the operator on the host. The value can be an absolute path or a relative path.	Yes
-out, --output	Sets the output path. The path can be either absolute or relative. The user who runs the tool must have the read and write permissions on the path. If this option is not specified, the outputs are generated to the current path where the command is executed.	No
-m, --model	Sets the path of the TensorFlow model file. The path can be either absolute or relative. If this option is set, the tool obtains the shape of the input layer from the TensorFlow model file and automatically dumps the shape, dtype, and attribute values of the operator in the operator information library definition file. If the dumped results are within the valid ranges configured in the operator information library definition file, they are automatically copied to the generated operator test case definition file. Otherwise, an error is reported. NOTICE: To use this option, TensorFlow 1.15 or 2.6.5 is required.	No
-q, --quiet	Enables quiet generation, which applies only to the -m option. If the -q option is not included, the system prompts you to modify the top-layer shape in the obtained model. If the -q option is included, the system does not prompt you to modify the top-layer shape.	No
-h, --help	Outputs the help information.	No

Generate and execute test cases.

**Table 2** Parameters for generating and executing test cases
Parameter	Description	Required
run	Generates operator ST cases.	Yes
-i, --input	Sets the path of the operator test case definition file (*.json). The path can be either absolute or relative. NOTE: A JSON file supports a maximum of 1000 cases.	Yes
-soc, --soc_version	Sets Ascend AI Processor version. NOTE: Products except for Atlas A3 training products/Atlas A3 inference products: Run the npu-smi info command on the server where the Ascend AI Processor is installed to obtain the Chip Name information. The actual value is AscendChip Name. For example, if Chip Name is xxxyy, the actual value is Ascendxxxyy. If Ascendxxxyy is the code sample path, you need to set ascendxxxyy. Atlas A3 training products/Atlas A3 inference products: Run the npu-smi info -t board -i id -c chip_id command on the server where the Ascend AI Processor is installed to obtain Chip Name and NPU Name. The actual value is Chip Name_NPU Name. For example, if the value of Chip Name is Ascendxxx and the value of NPU Name is 1234, the actual value is Ascendxxx_1234. If Ascendxxx_1234 is the code sample path, you need to set ascendxxx_1234. Note that: id: device ID, which is the NPU ID obtained by running the npu-smi info -l command. *chip_id: chip ID, which is obtained by running the npu-smi info -m* command.	Yes
-out, --output	Sets the output path. The path can be either absolute or relative. The user who runs the tool must have the read and write permissions on the path. If this option is not specified, the outputs are generated to the current path where the command is executed.	No
-c, --case_name	Set this option to the case (or cases separated by commas) to be executed. If it is set to all or not specified, all cases are executed.	No
-d, --device_id	Sets the NPU device ID, that is, the ID of Ascend AI Processor that runs ST cases. Defaults to 0.	No
-err_thr, --error_threshold	Customizes the accuracy standard. The value is a list containing two elements, formatted as "[threshold1,threshold2]". threshold1: threshold of the error between the operator output result and the benchmark data. If the actual error is greater than this threshold, the operator output result is recorded as the error data. threshold2: threshold of the ratio of error data to all data. If the actual ratio is less than this threshold, the accuracy is deemed acceptable. Otherwise, the precision does not meet the requirement. Defaults to "[0.01, 0.05]". Element value range: "[0.0, 1.0]". NOTE: The configured list must be enclosed in quotation marks ("") to avoid problems (for example: -err_thr "[0.01, 0.05]"). If this parameter is set both in the JSON file of the test case and in the corresponding msOpST command, the latter value is used for comparison. If this parameter is not set in neither of them, the default precision standard [0.01, 0.05] set in the msOpST command is used for comparison.	No
-conf, --config_file	Sets the path of the configuration file (msopst.ini) that implements advanced ST features. The path can be either absolute or relative. You can enable the following advanced features in the msopst.ini configuration file: Editable ST source code. Executable ST source code that has been edited. Host log level setting. Switch of log printing to the console. ATC log level setting. OS type and architecture of the operating environment for ATC model conversion. Model accuracy setting. Readable profile data of the operator running on Ascend AI Processor. If the --config_file file is not configured, the model forcibly uses the FP16 precision. For details about the msopst.ini configuration file, see Table 1.	No
-err_report, --error_report	Obtains the inconsistent results between the expected operator output and the actual output for failed ST cases. Defaults to false. true: generates error reports for failed ST cases, for example, the {*case.name}_error_report.csv* file. false: No error report is generated. NOTE: If this option is set to true, an independent CSV file is generated for each case based on case_name. The files are stored in the *{output_path}/{time_stamp}/*{op_type}/run/out/test_data/data/st_error_reports directory. A single .csv file can contain a maximum of 50,000 lines of data. If the number of lines exceeds 50,000, new .csv files are generated in sequence. The file name is in the format of {case.name}_error_report**0.csv.	No
-h, --help	Outputs the help information.	No

Generate single-operator board test framework.

**Table 3** Parameters for generating single-operator board test framework
Parameter	Description	Required
ascendc_test	Generates the board test code for the Ascend C operator to call the kernel function.	Yes
-i, --input	Sets the path of the operator test case definition file (.json file). The path can be either absolute or relative. NOTE: The specified operator ST case definition file (.json file) supports only one test case. Multiple types, formats, and shapes cannot be configured in the test case.	Yes
-kernel, --kernel_file	Sets the path of the kernel implementation file (*.cpp file) of the Ascend C operator. The path can be an absolute or relative path.	Yes
-out, --output	Output path of the test framework code. The value can be an absolute path or a relative path, and the user who executes the tool has the read and write permissions.	No
-h, --help	Outputs the help information.	No

Supplementary Notes

For details about other parameters of the msOpST tool, see Table 4.

**Table 4** Parameters
Parameter	Description	Description
get_shape	Obtains the shape.	Machine-machine interface, which can be ignored by users.
change_shape	Modifies the shape.
gen	Generates acl_op.json.
gen_testcase	Generates test files and data.
compare	Compares the results.
compare_by_path	Compares the results of files in specified paths.

Parent topic: msOpST (Operator Test)