Generating and Executing Test Cases

export DDK_PATH=${INSTALL_DIR}
export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

• Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.
• In {arch-os}, arch indicates the OS architecture, and os indicates the OS.

msopst run -i {**.json} -soc {Soc Version} -out {output path} -c {case name} -d {device id} -conf {msopst.ini path} -err_thr "[threshold1,threshold2]"

• The msopst.ini file is available in ${INSTALL_DIR}/python/site-packages/bin/.
• The following table describes the parameters in the msopst.ini file.
  

  The msopst.ini file uses the FP16 precision mode by default. To use other precision modes, manually modify the --precision_mode parameter of atc_singleop_advance_option in Table 1.

  Table 1 msopst.ini file description

  Parameter	Value	Description
  only_gen_without_run	True False (default)	Sets the run mode of the msOpST tool. For details, see Table 2.
  only_run_without_gen	True False (default)
  performance_mode	True False	Obtains the operator performance mode. If the setting is True, a series of performance result files are generated to the run/out/prof/JOBxxx/summary directory after the execution is successful. You can view the op_summary_0_1.csv file for details. To use this function, configure the environment variable of the CANN installation path. 1 export install_path=${INSTALL_DIR}
  ASCEND_GLOBAL_LOG_LEVEL	0: debug 1: info 2: warning 3 (default): error 4: null (no log output)	Host log level setting.
  ASCEND_SLOG_PRINT_TO_STDOUT	0 (default): disabled 1: enabled	Enables log printing to the screen.
  atc_singleop_advance_option	--log: sets the levels of logs to output. debug: outputs debug, info, warning, error, and event logs. info: outputs info, warning, error, and event logs. warning: outputs warning, error, and event logs. error (default): outputs error and event logs. null: no log output. --precision_mode: sets the precision mode. force_fp16 (default): forces float16 for operators supporting both float16 and float32. force_fp32: forces float32 for operators supporting both float16 and float32. allow_fp32_to_fp16: preserves the original precision for operators supporting float32; else, forces float16. must_keep_origin_dtype: preserves the original precision. allow_mix_precision: uses mixed precision. --host_env_os: sets the OS type. linux: Linux OS --host_env_cpu: sets the OS architecture type. x86_64: x86_64 architecture aarch64: AArch64 architecture Example: atc_singleop_advance_option="--log=info --host_env_os=linux --host_env_cpu=aarch64 --precision_mode=force_fp16"	Sets the advanced options for single-operator model conversion. If the OS and architecture of the model compilation environment are inconsistent with those of the model operating environment, set the OS type of the model operating environment using --host_env_os and --host_env_cpu. If this option is not set, the OS architecture of the model compilation environment where the ATC is located is used by default.
  HOST_ARCH	X86_64: x86_64 architecture aarch64: Arm64 architecture Example: HOST_ARCH="aarch64"	Architecture of the execution machine. Generally, this parameter is set in the separate deployment scenario.
  TOOL_CHAIN	g++ path: path of the g++ toolchain For example: TOOL_CHAIN="/usr/bin/g++"	Path of the C++ compiler. The value ends with g++. Generally, this parameter is set in the separate deployment scenario.

  Table 2 Run modes of msOpST

  #	only_gen_without_run	only_run_without_gen	Run Mode
  1	False	False	Generates and runs ST code.
  2	True	True/False	Generates ST code only.
  3	False	True	Runs ST code only.

• Examples:
  • Disable the advanced functions of the msOpST tool and run the following command to generate and execute ST cases:

    msopst run -i xx/AddCustom_case_timestamp.json -soc {Soc Version} -out ./output

  • The following is an example to generate ST cases only, with the advanced features of msOpST enabled, and then run the ST cases after manual modification.
    1. Run the following command to edit the msopst.ini file:

       1	vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

       Set only_gen_without_run and only_run_without_gen to enable msopst run mode 2. For details, see Table 2. Generates ST code only.

    2. Run the following command to generate the ST source code.

       msopst run -i xx/AddCustom_case_timestamp.json -soc {Soc Version} -out ./output -conf xx/msopst.ini

       Set -conf to the actual path of the msopst.ini configuration file.

       Modify the test code as needed after the ST cases are generated.

    3. Modify the msopst.ini file to change the run mode to running ST cases only.

       Run the following command to edit the msopst.ini file:

       vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

       Set only_gen_without_run and only_run_without_gen to enable msOpST run mode 3. For details, see Table 2. Runs ST code only.

    4. Run the edited ST source code.

       msopst run -i xx/AddCustom_case_timestamp.json -soc {Soc Version} -out ./output -conf xx/msopst.ini

    

    If the execution fails, analyze the cause as follows:

    • Refer to the description of aclError in aclError.
    • For details, see error code reference.
    • Check the log by following Log Reference.

• In the run mode that msopst generates the ST code only, a timestamp directory is generated in the directory specified by -out, and under the timestamp directory, a folder named after OpType is generated for storing ST cases. The directory structure is as follows:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

  {time_stamp} │ ├── OpType │ │ ├── CMakeLists.txt // Compilation script │ │ ├── inc // Directory of header files required by the test case code │ │ │ └── common.h │ │ │ └── op_execute.h │ │ │ └── op_runner.h │ │ │ └── op_test_desc.h │ │ │ └── op_test.h │ │ ├── run // Directory related to test case execution │ │ │ └── out │ │ │ └── test_data │ │ │ └── config │ │ │ └── acl.json // File for AscendCL initialization. Keep it intact. │ │ │ └── acl_op.json // Operator description file used to build a single-operator model │ │ │ └── data │ │ │ └── expect │ │ │ └── Test_xxx.bin │ │ ├── src │ │ │ └── CMakeLists.txt // Compilation script │ │ │ └── common.cpp // Common function file, used to read binary files │ │ │ └── main.cpp // Code for initializing and executing the operator test cases │ │ │ └── op_execute.cpp // Encapsulated AscendCL API for single-operator execution │ │ │ └── op_runner.cpp // Encapsulated API for loading the single-operator model file for execution │ │ │ └── op_test.cpp // Test class of the operator │ │ │ └── op_test_desc.cpp // File for loading and reading the operator test case information │ │ │ └── testcase.cpp // Test case definition file

• In the run mode that msopst generates and runs ST code, the execution result is printed to the screen after the command is executed. Then, a timestamp directory is generated in the directory specified by -out, and under the timestamp directory, a folder named after OpType is generated for storing both the ST cases and the execution results. The directory structure is as follows:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

  {time_stamp} │ ├── OpType │ │ ├── build │ │ │ └── intermediates // Directory of intermediate files generated at compile time │ │ │ └── xxx │ │ ├── CMakeLists.txt // Compilation script │ │ ├── inc │ │ │ ├── common.h │ │ │ ├── op_execute.h │ │ │ ├── op_runner.h │ │ │ ├── op_test_desc.h │ │ │ └── op_test.h │ │ ├── run // Directory related to test case execution │ │ │ └── out │ │ │ ├── fusion_result.json │ │ │ ├── main // Directory of executable files of the operator test cases │ │ │ ├── op_models // Directory of single-operator offline model files │ │ │ ├── xx.om │ │ │ ├── result_files │ │ │ ├── result.txt │ │ │ ├── Test_xxx_output_x.bin // Result binary generated after test case execution │ │ │ └── test_data // Directory for storing test data files │ │ │ ├── config │ │ │ ├── acl_op.json // Operator description file used to build a single-operator model │ │ │ ├── acl.json // File for AscendCL initialization. Keep it intact. │ │ │ ├── data // Constructed test data │ │ │ ├──expect │ │ │ ├──Test_xxxx.bin // Binary of the expected output │ │ │ ├──st_error_reports │ │ │ ├──Test_xxxx.csv // Used to store inconsistent data │ │ │ ├──Test_xxxx.bin // Test data binary │ │ └── src │ │ ├── CMakeLists.txt // Compilation script │ │ ├── common.cpp // Common function file, used to read binaries │ │ ├── main.cpp // Code for initializing and executing the operator test cases │ │ ├── op_execute.cpp // Encapsulated AscendCL API for single-operator execution │ │ ├── op_runner.cpp // Encapsulated API for loading the single-operator model file for execution │ │ ├── op_test.cpp // Test class of the operator │ │ ├── op_test_desc.cpp // File for loading and reading the operator test case information. │ │ └── testcase.cpp // Test case definition file │ └── st_report.json // Execution report

  After the command is executed successfully, an st_report.json report is automatically generated. The report records the detailed test information and the runtime states in each phase, which facilitates troubleshooting and test result comparison. The save path of st_report.json is that followed "The st_report saved in", as shown in Figure 1.

  Figure 1 Result example
  

  Table 3 st_report.json description

  Field			Description
  run_cmd	-	-	Command
  report_list	-	-	List of reports of test cases.
  	trace_detail	-	Execution details.
  		st_case_info	Test information, including the following: expect_data_path: path of the expected result case_name: test case name input_data_path: path of input data planned_output_data_paths: path of the actual result op_params: operator parameter information
  		stage_result	Result information in each runtime phase, including the following: status: running status of the phase, either success or failure result: output result stage_name: stage name cmd: command
  	case_name	-	Test name
  	status	-	Actual test result, either success or failure
  	expect	-	Expected test result, either success or failure
  summary	-	-	Summary of comparison of the actual test result and the expected test result.
  	test case count	-	Number of test cases
  	success count	-	Number of test cases whose result is the same as the expected result
  	failed count	-	Number of test cases whose result differs from the expected result

1. The AscendCL APIs are used to load and execute the single-operator model file during ST case execution. As such, you need to configure the environment variables required for AscendCL application building in the development environment based on the operating environment architecture.
   • When the architecture of the development environment is the same as that of the operating environment, the environment variables are as follows:

     export DDK_PATH=${INSTALL_DIR}
     export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

   • When the architecture of the development environment is different from that of the operating environment, the environment variables are as follows:

     export DDK_PATH=${INSTALL_DIR}/{arch-os}
     export NPU_HOST_LIB=${INSTALL_DIR}/{arch-os}/devlib

   

   • Replace ${INSTALL_DIR} with the actual CANN component directory. If the Ascend-CANN-Toolkit package is installed as the root user, the CANN component directory is /usr/local/Ascend/ascend-toolkit/latest.
   • In <arch-os>, arch and os indicate the OS architecture and OS of the operating environment, respectively.

1. Run the following command to edit the msopst.ini file:

   vim ${INSTALL_DIR}/python/site-packages/bin/msopst.ini

2. Set only_gen_without_run and only_run_without_gen to enable msopst run mode 2. For details, see Table 2. Generates ST code only.
3. If the architectures of the development environment and operating environment are different, change the values of HOST_ARCH and TOOL_CHAIN by referring to Table 1.
4. Run the following command to generate the ST source code.

   msopst run -i xx/AddCustom_case_timestamp.json -soc {Soc Version} -out {output path} -conf xx/msopst.ini

   Set -conf to the actual path of the msopst.ini configuration file.

   Modify the test code as needed after the ST cases are generated.

5. After the execution is complete, ST cases are generated in {output path}, and an executable file main is generated by using the g++ compiler. In addition, the number of test cases that are executed this time and the execution result are displayed. The st_report.json report is generated and saved in the directory specified by the path in the "The st report saved in" message. For details about the report, see Table 3.

1. Copy the out folder in the run directory of the operator project directory in the development environment to any directory in the operating environment, for example, ${INSTALL_DIR}/Ascend_project/run_add/.
2. Run the executable file in the out folder in the operating environment.
   Go to the parent directory of the out folder and run the following commands:

   chmod +x main
   ./main

After the execution is complete, the execution result of the test cases is displayed, as shown in Figure 2.

Figure 2 Execution result