Generating and Executing Test Cases

export DDK_PATH=${INSTALL_DIR}
export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

• Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.
• In {arch-os}, arch indicates the OS architecture, and os indicates the OS.

./msopst run -i {**.json}  -soc {Soc Version}  -out {output path} -c {case name} -d {device id} -conf {msopst.ini path} -err_thr "[threshold1,threshold2]"

• The msopst.ini file is available in ${INSTALL_DIR}/python/site-packages/bin/.

  Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.

• The following table describes the parameters in the msopst.ini file.

  Table 2 msopst.ini file description

  Field	Value	Description
  only_gen_without_run	True False (default)	Sets the run mode of the msopst tool. For details, see Table 3.
  only_run_without_gen	True False (default)
  performance_mode	True False	Obtains the operator performance mode. If the setting is True, a series of performance result files are generated to the run/out/prof/JOBxxx/summary directory after the execution is successful. You can view the op_summary_0_1.csv file for details. To use this function, configure the environment variable of the CANN installation path. export install_path=/home/HwHiAiUser/Ascend/ascend-toolkit/latest
  ASCEND_GLOBAL_LOG_LEVEL	0: debug 1: info 2: warning 3 (default): error 4: null (no log output)	Sets the host log level.
  ASCEND_SLOG_PRINT_TO_STDOUT	0 (default): disabled 1: enabled	Enables log printing to the screen.
  atc_singleop_advance_option	--log: sets the levels of logs to output. debug: outputs debug, info, warning, error, and event logs. info: outputs info, warning, error, and event logs. warning: outputs warning, error, and event logs. error (default): outputs error and event logs. null: no log output. --precision_mode: sets the precision mode. force_fp16 (default): forces float16 for operators supporting both float16 and float32. force_fp32: forces float32 for operators supporting both float16 and float32. allow_fp32_to_fp16: preserves the original precision for operators supporting float32; else, forces float16. must_keep_origin_dtype: preserves the original precision. allow_mix_precision: uses mixed precision. --host_env_os: sets the OS type. linux: Linux OS --host_env_cpu: sets the OS architecture type. x86_64: x86_64 architecture aarch64: AArch64 architecture Example: atc_singleop_advance_option = "--log=info --host_env_os=linux --host_env_cpu=aarch64 --precision_mode=force_fp16"	Sets the advanced options for single-operator model conversion. If the OS and architecture of the model build environment are inconsistent with those of the model operating environment, set the OS type of the model operating environment using --host_env_os and --host_env_cpu. If this option is not set, the OS architecture of the model compilation environment where the ATC is located is used by default.
  HOST_ARCH	X86_64: x86_64 architecture aarch64: Arm64 architecture Example: HOST_ARCH="aarch64"	Architecture of the execution machine. Generally, this parameter is set in the separate deployment scenario.
  TOOL_CHAIN	g++ path: path of the g++ toolchain Example: TOOL_CHAIN="/usr/bin/g++"	Path of the C++ compiler. The value ends with g++. Generally, this parameter is set in the separate deployment scenario.

  Table 3 Run modes of msopst

  #	only_gen_without_run	only_run_without_gen	Run Mode
  1	False	False	Generates and runs ST code.
  2	True	True/False	Generates ST code only.
  3	False	True	Runs ST code only.

• Examples:
  • The following is an example to go to the directory where the msopst tool is located and generate ST cases and execute them, with the advanced features of msopst disabled:

    ./msopst run -i xx/Add_case_timestamp.json -soc {Soc Version} -out ./output

  • The following is an example to generate ST cases only, with the advanced features of msopst enabled, and then run the ST cases after manual modification.
    1. Run the following command to edit the msopst.ini file:

       vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

       Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.

       Set only_gen_without_run and only_run_without_gen to enable msopst run mode 2. For details, see Table 3. In this mode, msopst generates ST code only.

    2. Go to the directory where the msopst tool is located and run the following command to generate the ST source code:

       ./msopst run -i xx/Add_case_timestamp.json -soc {Soc Version} -out ./output -conf xx/msopst.ini

       Set -conf to the actual path of the msopst.ini configuration file.

       Modify the test code as needed after the ST cases are generated.

    3. Modify the msopst.ini file to change the run mode to running ST cases only.

       Run the following command to edit the msopst.ini file:

       vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

       Set only_gen_without_run and only_run_without_gen to enable msopst run mode 3. For details, see Table 3. In this mode, msopst runs ST code only.

    4. Go to the directory where msopst is located and then run the following command to run the modified ST source code:

       ./msopst run -i xx/Add_case_timestamp.json -soc {Soc Version} -out ./output -conf xx/msopst.ini

    

    If the execution fails, perform the following operations:

    • Refer to the description of aclError in "AscendCL API Reference (C&C++) > Data Types and Operations" of CANN AscendCL Application Software Development Guide (C&C++).
    • For details, see "Error Codes" in the Troubleshooting.
    • Check the log by following Log Reference.

• In the run mode that msopst generates the ST code only, a timestamp directory is generated in the directory specified by -out, and under the timestamp directory, a folder named after OpType is generated for storing ST cases. The directory structure is as follows:

   {time_stamp}
  │   ├── OpType
  │   │   ├── CMakeLists.txt            // Build script
  │   │   ├── inc                       // Directory of header files required by the test case code
  │   │   │   └── common.h
  │   │   │   └── op_execute.h
  │   │   │   └── op_runner.h
  │   │   │   └── op_test_desc.h
  │   │   │   └── op_test.h
  │   │   ├── run                       // Directory related to test case execution
  │   │   │   └── out
  │   │   │       └── test_data
  │   │   │          └── config
  │   │   │             └── acl.json      // File for AscendCL initialization. Keep it intact.
  │   │   │             └── acl_op.json   // Operator description file used to build a single-operator model
  │   │   │          └── data
  │   │   │             └── expect
  │   │   │             └── Test_xxx.bin
  │   │   ├── src
  │   │   │   └── CMakeLists.txt    // Build script
  │   │   │   └── common.cpp         // Common function file, used to read binary files
  │   │   │   └── main.cpp            // Code for initializing and executing the operator test cases
  │   │   │   └── op_execute.cpp        // Encapsulated AscendCL API for single-operator execution
  │   │   │   └── op_runner.cpp         // Encapsulated API for loading the single-operator model file for execution
  │   │   │   └── op_test.cpp          // Test class of the operator
  │   │   │   └── op_test_desc.cpp      // File for loading and reading the operator test case information
  │   │   │   └── testcase.cpp             // Test case definition file

• In the run mode that msopst generates and runs ST code, the execution result is printed to the screen after the command is executed. Then, a timestamp directory is generated in the directory specified by -out, and under the timestamp directory, a folder named after OpType is generated for storing both the ST cases and the execution results. The directory structure is as follows:

   {time_stamp}
  │   ├── OpType
  │   │   ├── build
  │   │   │   └── intermediates             // Directory of intermediate files generated at build time
  │   │   │       └── xxx
  │   │   ├── CMakeLists.txt    // Build script
  │   │   ├── inc
  │   │   │   ├── common.h
  │   │   │   ├── op_execute.h
  │   │   │   ├── op_runner.h
  │   │   │   ├── op_test_desc.h
  │   │   │   └── op_test.h
  │   │   ├── run                           // Directory related to test case execution
  │   │   │   └── out
  │   │   │       ├── fusion_result.json
  │   │   │       ├── main           // Directory of executable files of the operator test cases
  │   │   │       ├── op_models         // Directory of single-operator offline model files
  │   │   │          ├── xx.om
  │   │   │       ├── result_files
  │   │   │          ├── result.txt
  │   │   │          ├── Test_xxx_output_x.bin   // Result binary generated after test case execution
  │   │   │       └── test_data         // Directory for storing test data files
  │   │   │          ├── config
  │   │   │             ├── acl_op.json    // Operator description file used to build a single-operator model
  │   │   │             ├── acl.json       // File for AscendCL initialization. Keep it intact.
  │   │   │          ├── data                // Constructed test data
  │   │   │             ├──expect
  │   │   │                 ├──Test_xxxx.bin      // Binary of the expected output
  │   │   │             ├──st_error_reports
  │   │   │                 ├──Test_xxxx.csv       // Used to store inconsistent data
  │   │   │             ├──Test_xxxx.bin      // Test data binary
  │   │   └── src
  │   │       ├── CMakeLists.txt    // Build script
  │   │       ├── common.cpp         // Common function file, used to read binaries
  │   │       ├── main.cpp            // Code for initializing and executing the operator test cases
  │   │       ├── op_execute.cpp        // Encapsulated AscendCL API for single-operator execution
  │   │       ├── op_runner.cpp         // Encapsulated API for loading the single-operator model file for execution
  │   │       ├── op_test.cpp          // Test class of the operator
  │   │       ├── op_test_desc.cpp      // File for loading and reading the operator test case information.
  │   │       └── testcase.cpp             // Test case definition file
  │   └── st_report.json        // Execution report

  After the command execution is successful, the st_report.json report is generated and saved in the path following "The st_report saved in", as shown in Figure 1. The report records the test information and the running status of each phase.

  In addition, st_report.json can be used to compare test results. If a fault occurs during the running, you can query the running information based on the report to locate the fault.

  Figure 1 Result example
  

  Table 4 st_report.json description

  Field			Description
  run_cmd	-	-	Command.
  report_list	-	-	List of reports of test cases.
  	trace_detail	-	Execution details.
  		st_case_info	Test information, including the following: expect_data_path: path of the expected result case_name: test case name input_data_path: path of input data planned_output_data_paths: path of the actual result op_params: operator parameter information
  		stage_result	Result information in each runtime phase, including the following: status: running status of the phase, either success or failure result: output result stage_name: stage name cmd: command
  	case_name	-	Test name.
  	status	-	Actual test result, either success or failure.
  	expect	-	Expected test result, either success or failure.
  summary	-	-	Summary of comparison of the actual test result the expected test result.
  	test case count	-	Number of test cases.
  	success count	-	Number of test cases whose result is the same as the expected result.
  	failed count	-	Number of test cases whose result differs from the expected result.

1. The CANN development kit should be installed based on the operating environment architecture. For details, see Environment Setup. (If the architecture of the development environment is the same as that of the operating environment, you do not need to install the CANN development kit again.)
2. The AscendCL APIs are used to load and execute the single-operator model file during ST case execution. As such, you need to configure the environment variables required for AscendCL application building in the development environment based on the operating environment architecture.
   • When the architecture of the development environment is the same as that of the operating environment, the environment variables are as follows:

     export DDK_PATH=${INSTALL_DIR}
     export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

   • When the architecture of the development environment is the different from that of the operating environment, the environment variables are as follows:

     export DDK_PATH=${INSTALL_DIR}/{arch-os}
     export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

   

   • Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.
   • In <arch-os>, arch and os indicate the architecture and OS of the operating environment, respectively.

1. Run the following command to edit the msopst.ini file:

   vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

   Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.

2. Set only_gen_without_run and only_run_without_gen to enable msopst run mode 2. For details, see Table 3. In this mode, msopst generates ST code only.
3. If the architectures of the development environment and operating environment are different, change the values of HOST_ARCH and TOOL_CHAIN by referring to Table 2.
4. Go to the directory where the msopst tool is located and run the following command to generate the ST source code:

   ./msopst run -i {**.json} -soc {Soc Version} -out {output path} -conf xx/msopst.ini

   Set -conf to the actual path of the msopst.ini configuration file. Modify the test code as needed after the ST cases are generated.

5. After the execution is complete, ST cases are generated in {output path}, and an executable file main is generated by using the g++ compiler. In addition, the number of test cases that are executed this time and the execution result are displayed. The st_report.json report is generated and saved in the directory specified by the path in the "The st report saved in" message. For details about the report, see Table 4.

1. Copy the out folder in the run directory of the operator project directory in the development environment to any directory in the operating environment, for example, /home/HwHiAiUser/Ascend_project/run_add/.
2. Run the executable file in the out folder in the operating environment.
   Go to the parent directory of the out folder and run the following commands:

   chmod +x main
   ./main

After the execution is complete, the execution result of the test cases is displayed, as shown in Figure 2.

Figure 2 Execution result