Documentation

TBE Operator UT

Prerequisites

The custom operator has been developed, including the operator implementation code and operator prototype definition. For details, see Operator Code Implementation (TBE DSL) and Operator Prototype Definition.

CentOS 7.8 Arm containers do not support the UT of the operator implementation code.

Generating UT Case Files

  1. Create UT cases.
    1. A UT case file can be created in either of the following ways:

      Right-click the root directory of the operator project and choose New Cases > TBE UT Case from the shortcut menu.

      If UT cases of the operator exist, right-click the testcases or testcases > ut directory, and choose New Cases > TBE UT Case from the shortcut menu to create UT cases.

    2. In the Create UT for an Operator window, choose the target operator, and click OK, as shown in the following figure.

      If the UT cases of the operator already exist, the message "testcases/ut/ops_test/xx already exists. Do you want to overwrite?" is displayed.

      You can choose Overwrite to overwrite the current test cases or Cancel to cancel the overwriting.

      After the creation is complete, the testcases folder is generated in the root directory of the operator project. The directory structure is as follows:

      ├── MyOperator           // Root directory of the project
│   ├──  testcases                            
│   │   ├── libs                 // GTest framework. It is a third-party dependency and can be ignored.
│   │   ├──  ut                              
│   │   │   ├── ops_test
│   │   │   │   ├── add   
│   │   │   │   │   ├── CMakeLists.txt       // Build script
│   │   │   │   │   ├── test_add_impl.py     // Test case file for the operator implementation code
│   │   │   │   │   ├── test_add_proto.cc   // Test case file for the operator prototype definition code
│   │   │   │   ├── CMakeLists.txt            // Build script
│   │   │   │   ├── test_main.cc             // Main entry for test case calling
│   │   │   ├ CMakeLists.txt
  2. During NeoKylin and Kylin OS running, add the following information in bold to the testcases/ops_test/add/CMakeLists.txt file. For other OSs, skip this step.
    add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)
set(CMAKE_CXX_FLAGS "-std=c++11")   
set(PROJECT_DIR "$ENV{PROJECT_PATH}")
set(GTEST_DIR ${PROJECT_DIR}/testcases/libs/gtest)
set(ADK_DIR "$ENV{ADK_PATH}")
set(ATC_DIR ${ADK_DIR}/atc)
set(OP_PROTO_SRC_DIR ${PROJECT_DIR}/op_proto)

message(STATUS "ATC_DIR=${ATC_DIR}")

enable_testing()

include_directories(
        "${GTEST_DIR}/include"
        "${ATC_DIR}/include"
        "${OP_PROTO_SRC_DIR}"
        )

aux_source_directory(${OP_PROTO_SRC_DIR} OP_PROTO_SOURCE_SRCS)
file(GLOB OP_PROTO_TEST_FILES **proto.cc)

link_directories(
        "${ATC_DIR}/lib64"
        "${GTEST_DIR}"
        "/usr/local/gcc7.3.0/lib64/" 
)

set(CUSTOM_OBJECT_NAME "add_proto_test")

add_executable(${CUSTOM_OBJECT_NAME}
        ${PROJECT_DIR}/testcases/ut/ops_test/test_main.cc ${OP_PROTO_SOURCE_SRCS} ${OP_PROTO_TEST_FILES})

target_link_libraries(${CUSTOM_OBJECT_NAME} gtest c_sec alog pthread error_manager graph register)

    The GCC version must be 7.5.0 or later for the UT. If the GCC version does not meet the requirement, upgrade the GCC.

  3. Write UT cases in Python for the operator implementation code.
    In the testcases/ut/ops_test/add/test_add_impl.py file, directly write the UT cases. 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
    		 
    import ...
from op_test_frame.ut import BroadcastOpUT   # Import the UT class based on the operator type.

ut_case = BroadcastOpUT("add")      # Instantiate the UT case. ut_case is a keyword of the UT framework and cannot be modified. add is the type of the operator.

def calc_expect_func(input_x, input_y, output_z):    # Define the function for generating the expected data.
    res = input_x["value"] + input_y["value"]
    return [res, ]    # Return the expected data.

# Add test cases.
ut_case.add_precision_case("all", {
    "params": [{"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (32,), "shape": (32,),
                "param_type": "input"},
               {"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (32,), "shape": (32,),
                "param_type": "input"},
               {"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (32,), "shape": (32,),
                "param_type": "output"}],
    "calc_expect_func": calc_expect_func
})
# If multiple test cases are defined, multiple ut_case.add_precision_case functions need to be defined.
ut_case.add_precision_case("all", {
    "params": [{"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (16,2), "shape": (16,2),
                "param_type": "input"},
               {"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (16,2), "shape": (16,2),
                "param_type": "input"},
               {"dtype": "float16", "format": "ND", "ori_format": "ND", "ori_shape": (16,2), "shape": (16,2),
                "param_type": "output"}],
    "calc_expect_func": calc_expect_func
})
    1. Import the UT class based on the operator type. For details, see UT API Reference.
    2. Instantiate the UT cases. For details about how to use OpUT, see Definition of the OpUT Test Class.
    3. Define the function for generating the expected data.
    4. Add test cases.

      The fields and value ranges in params of a test case need to be determined based on the input parameters of the operator implementation file. The ori_shape and ori_format fields in the input tensor are optional. However, if the parameter validation decorator is used, these two fields are mandatory.

      For details about how to use each test API, see UT API Reference.

      To compare the result with the expectation, call add_precision_case.

  4. Write UT cases in C++ for the operator prototype definition.

    In the testcases/ut/ops_test/add/test_add_proto.cc file, write UT cases in C++ for the operator prototype definition. Define the operator instance, update the input and output of the operator, call InferShapeAndType, and verify the execution process as well as the result of InferShapeAndType.

    1. Include the GTest framework and the header file of the operator IR definition.

      C++ UT cases use the GTest framework. Therefore, you need to include the GTest framework. The operator prototype is defined in the prototype definition header file. Therefore, you need to include the .h file of the prototype definition.

      1
2
3
4
5
6
      		 
      // Import the GTest framework.
#include <gtest/gtest.h>
// Include the basic vector library.
#include <vector>
// Include the header file of the operator IR definition.
#include "add.h"
    2. Define a test class.

      C++ UT cases use the GTest framework. Therefore, a class needs to be defined to inherit the gtest class.

       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
      		 
      #include <gtest/gtest.h>
#include <vector>
#include "add.h"

class AddTest : public testing::Test {
protected:
    static void SetUpTestCase() {
        std::cout << "add test SetUp" << std::endl;
}

    static void TearDownTestCase() {
        std::cout << "add test TearDown" << std::endl;
    }
};

      The test class name can be user-defined and suffixed with Test.

    3. Write test cases.

      Write a test case function for each scenario. In this case, an operator instance needs to be constructed, including the operator name, shape, and data type. Call the InferShapeAndType function and compare the inferred shape and dtype with the expected results.

      The following is an example:

       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
      		 
      TEST_F(AddTest, add_test_case_1) {
    // Define the operator instance, input shape, and input type, which are carried by the TensorDesc instance.
    ge::op::Add add_op;  // Add is the operator type, which must be the same as the OpType argument in REG_OP(OpType) in the operator definition file.
    ge::TensorDesc tensorDesc;
    ge::Shape shape({2, 3, 4});
    tensorDesc.SetDataType(ge::DT_FLOAT16);
    tensorDesc.SetShape(shape);
    tensorDesc.SetOriginShape(shape);

    // Update the operator inputs. The input name must be the same as that in the prototype definition file (.h). For example, x1 and x2 are inputs of the Add operator.
    add_op.UpdateInputDesc("x1", tensorDesc);
    add_op.UpdateInputDesc("x2", tensorDesc);
    // Call the InferShapeAndType function. The InferShapeAndType() API is a fixed API. During case execution, the shape inference function in the operator prototype definition is automatically called.
    auto ret = add_op.InferShapeAndType();
    // Verify whether the calling process is successful.
    EXPECT_EQ(ret, ge::GRAPH_SUCCESS);  

    // Obtain the operator output and compare the shape and type. The name of the operator output must be the same as that in the prototype definition file (.h). For example, the operator output is y.
    auto output_desc = add_op.GetOutputDesc("y");
    EXPECT_EQ(output_desc.GetDataType(), ge::DT_FLOAT16);
    std::vector<int64_t> expected_output_shape = {2, 3, 4};
    EXPECT_EQ(output_desc.GetShape().GetDims(), expected_output_shape);
}

      If the shapes of inputs are different, define multiple TensorDesc objects, as follows:

       1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
      		 
          ge::op::Operator1 operator1_op;  // Operator1 indicates the operator type.
    ge::TensorDesc tensorDesc1;
    ge::TensorDesc tensorDesc2;
    ge::Shape shape1({2, 3, 4});
    ge::Shape shape2({3, 4, 5});
    tensorDesc1.SetDataType(ge::DT_FLOAT16);
    tensorDesc1.SetShape(shape1);
    tensorDesc1.SetOriginShape(shape1);
    tensorDesc2.SetDataType(ge::DT_FLOAT16);
    tensorDesc2.SetShape(shape2);
    tensorDesc2.SetOriginShape(shape2);

    // Update the operator inputs.
    operator1_op.UpdateInputDesc("x1", tensorDesc1);
    operator1_op.UpdateInputDesc("x2", tensorDesc2);

Running UT Cases

  1. Run the UT cases of the operator implementation file.
    You can run the UT cases of all operators in the current project or run the UT cases of a single operator.
    • Right-click the testcases/ut/ops_test folder and choose Run Tbe Operator 'All' UT Impl with coverage from the shortcut menu to run the test cases of operator implementation code in the folder.
    • Right-click the testcases/ut/ops_test/operator_name folder and choose Run Tbe Operator 'operator_name' UT Impl with coverage to run test cases of implementation code of a single operator.
    • Right-click the testcases/ut/ops_test/operator_name/test_operator_name_impl.py file and choose Run Tbe Operator'operator_name'UT Impl with coverage to run test cases of implementation code of a single operator.
    When the cases run for the first time, the run configuration page is displayed, as shown in Figure 1. Configure the parameters based on the actual requirements and click Run. For details about how to modify the run configurations, see Editing a Run Task Configuration.
    Figure 1 Run configurations
    Table 1 Run configuration

    Parameter

    Description

    Name

    Name of the run configuration (user-defined).

    Test Type

    ut_impl is recommended.

    Compute Unit

    Compute unit. Only AI Core and Vector Core are supported for TBE operators.

    SoC Version

    Current version of the Ascend AI Processor.

    Target

    Target operating environment.

    • Simulator_Function: functional simulation environment.
    • Simulator_Performance: performance simulation environment.
    • Simulator_TMModel: quickly displays the scheduling pipeline of operator execution. Actual operator computing is not performed. This function applies only to Ascend 310 AI Processors and Ascend 910 AI Processors.

    Operator Name

    Test cases to run.

    • all: all test cases
    • Other values: test cases of a specific operator

    Case Names

    Test cases to run, that is, the Python UT cases of the operator implementation code. You can select all or some test cases.
  2. Check the execution result.

    After the execution is complete, view the execution result in the Run log print window at the bottom.

    1. If the operator expectation function "calc_expect_func": calc_expect_func is configured, the comparison result will be printed in logs after the operator is successfully executed.
      • Error count: number of test cases whose error is greater than the absolute error (atol).
      • Max atol error count: number of test cases whose error is greater than the expected value multiplied by the maximum absolute error (max_atol)
      • Threshold count (rtol * data_size): maximum number of errors that are allowed to exceed the custom precision (relative error (rtol) x number of tensor elements)

      Test results:

      • If the value of Max atol error count is greater than 0, the UT fails.
      • If the value of Error count is greater than the value of Threshold count, the UT fails.
    2. In the Run window, click the URL in index.html to view the UT case coverage. In the URL, localhost indicates the server IP of MindStudio.

      To view the test results of UT cases, you need to use a browser. If no browser is available, install one.

      If "Page'http://***.html'requested without authorization, you can copy URL and open it in browser to trust it." is displayed, rectify the fault by referring to Analysis Sample of Timeline-based Tuning of AI CPU Operators.

    3. Click the operator on the HTML page to navigate to the UT case coverage dialog box, as shown in Figure 2. The green and red labels are used to indicate the coverage rates.
      Figure 2 UT coverage details

    4. View the outputs.
      • If Target is set to Simulator_Performance in the run configuration, you can view the execution pipeline in the profiling dialog box. See the following figure.

        Parallel Analysis: parallel analysis chart. The Parallel Analysis view displays the parallelism degree of each instruction in the run of a single-operator. The horizontal axis indicates the time (in clock ticks), and the vertical axis indicates the instruction units. Move the mouse to the time range, the time consumed by a single instruction and related instructions is displayed.

        • The duration between the Start and End indicates the time range for displaying data. End indicates the number of clock cycles. A larger value indicates a longer running time. You can scroll the mouse to change the time range for displaying data in the chart.
        • Right-click the time range. A shortcut menu is displayed, asking you to go to the profiling instruction flow view and the TBE, CCE, and Assembly source code menus.
      • If Target is set to Simulator_TMModel in the run configuration, you can view the execution pipeline, as shown in the following figure.

        If operators are developed in TIK mode, the MindStudio UT allows you to jump from the pipeline diagram to the TIK code and CCE code, helping you quickly locate the corresponding code position. For details, see Setting Code Redirection.

        Before the processor performs computation, Vector Unit, Cube Unit, MTE1, MTE2, MTE3, and more units are initialized. As a result, data is generated on each unit before data movement progress is displayed in the timeline.

  3. Run the UT cases defined in the operator prototype.

    You can run the UT cases of all operators in the current project or run the UT cases of a single-operator.

    • Right-click the testcases/ut/ops_test folder and choose Run Tbe Operator 'All' UT Proto from the shortcut menu to run the test cases of the operator prototype definition code in the folder.
    • Right-click the testcases/ut/ops_test/operator_name folder and choose Run Tbe Operator 'operator_name' UT Proto from the shortcut menu to run the test cases of prototype definition code of a single operator.
    • Right-click the testcases/ut/ops_test/operator_name/test_operator_name_proto.cc file and choose Run Tbe Operator 'operator_name' UT Proto from the shortcut menu to run the test cases of prototype definition code of a single operator.
    When the cases run for the first time, the corresponding configuration dialog box is displayed. Configure the parameters and click Run. For details about how to modify the run configurations, see Editing a Run Task Configuration.
    Table 2 Run configuration

    Parameter

    Description

    Name

    Name of the run configuration (user-defined).

    Test Type

    ut_proto is recommended.

    Compute Unit

    Compute unit. Only AI Core and Vector Core are supported for TBE operators.

    Operator Name

    Test cases to run.

    • all: all test cases
    • Other values: test cases of a specific operator

    Case Names

    Select test cases to be executed, that is, the test cases defined in TEST_F. You can select all or some test cases.
  4. Check the execution result.

    After the execution is complete, view the execution result in the log print window at the bottom. The result shows the test case count, including successful execution and failed execution, as shown in the following figure.

Performing OP Tiling UTs

Perform the following operations when OP Tiling is involved in the dynamic-shape operator project.

  1. Create an OP Tiling implementation file.
    1. Right-click the op_tiling folder and choose New > File from the shortcut menu.
    2. In the displayed New File page, enter {op_name}.cc. You can also customize the file name, as long as it is a .cc file. See the following figure.

    3. After you press Enter, an empty {op_name}.cc file is generated. You can write the file based on service requirements.
  2. Create OP Tiling UT cases.
    1. A UT case file can be created in any of the following ways:

      Right-click the root directory of the operator project and choose New Cases > TBE OP TILING UT Case from the shortcut menu.

      If UT cases of the operator project exist, right-click the testcases or testcases > ut directory, and choose New Cases > TBE OP TILING UT Case from the shortcut menu to create OP Tiling UT cases.

    2. On the displayed operator selection dialog box, select the operator for which the OP Tiling UT template project is generated and click OK. See the following figure.

      If the OP Tiling UT cases of the operator already exist, the message "testcases/ut/ops_tiling_test/xx already exists. Do you want to overwrite?" is displayed.

      You can click Overwrite or Cancel.

      After the creation is complete, the testcases folder is generated in the root directory of the operator project. The directory structure is as follows:

      ├── MyOperator           // Root directory of the project
│   ├──  testcases                            
│   │   ├── libs                 // GTest framework. It is a third-party dependency and can be ignored.
│   │   ├──  ut                              
│   │   │   ├── ops_tiling_test
│   │   │   │   ├──add
│   │   │   │   │   ├── CMakeLists.txt       // Build script
│   │   │   │   │   ├── test_add_tiling.cc      // Test case file of operator tiling
│   │   │   │   ├── CMakeLists.txt            // Build script
│   │   │   │   ├── test_main.cc             // Main entry for test case calling
  3. Run the OP Tiling UT cases.

    You can run the OP Tiling UT cases of all operators in the current project or run the OP Tiling UT cases of a single operator.

    • Right-click the testcases/ut/ops_tiling_test folder and choose Run TBE Operator'all'UT Tiling from the shortcut menu to run the test cases of operator prototype definition code in the folder.
    • Right-click the testcases/ut/ops_tiling_test/operator_name folder and choose Run TBE Operator'Name'UT Tiling from the shortcut menu to run test cases of the prototype definition code of a single operator.
    • Right-click the testcases/ut/ops_tiling_test/operator_name/test_operator_name_tiling_.cc file and choose Run TBE Operator'operator_name'UT Tiling from the shortcut menu to run test cases of prototype definition code of a single operator.
    When the cases run for the first time, the corresponding configuration dialog box is displayed. Configure the parameters and click Run. For details about how to modify the run configurations, see Editing a Run Task Configuration.
    Table 3 Run configuration

    Parameter

    Description

    Name

    Name of the run configuration (user-defined).

    Test Type

    Select ut_tiling.

    Compute Unit

    Compute unit. Select AI Core or Vector Core.

    Operator Name

    Test cases to run.

    • all: all test cases
    • Other values: test cases of a specific operator

    Case Names

    Select test cases to be executed, that is, the test cases defined in TEST_F.

    You can select all or some test cases.
  4. Check the execution result.

    After the execution is complete, view the execution result in the log print window at the bottom. The result shows the test case count, including successful execution and failed execution. See Figure 3.

    Figure 3 Execution result
Parent topic: UT