Developing a Plugin Framework

The plugin framework provided by Vision SDK defines how to generate standard plugins to implement data interconnection between plugins and set plugin properties. Vision SDK supports single-input, single-output, multi-input, and multi-output plugins.

1. Develop a plugin class.
   1. Inherit the MxPluginBase class in MxPluginBase.h and customize the plugin class name.
   2. Rewrite the three member functions Init(), DeInit(), and Process() in the base class. For details, see Table 1.
   3. MxpluginBase contains three common properties: status_, deviceId_, and dataSource_ (see Table 2), which can be configured by modifying the pipeline file.

      Table 1 Member functions

      Member Function	Description
      Init()	This API is called only once to obtain configuration parameters and initialize service logic (for example, allocate reusable memory in the plugin).
      DeInit()	This API is called only once to deinitialize a job (for example, releasing the memory).
      Process()	This API is driven by data and is used to receive data from upstream plugins and send results to the output port after the service processing logic of users is completed.

      Table 2 Common attributes of plugins

      Common Attribute	Description
      status_	Specifies whether the plugin runs in synchronous or asynchronous mode. The value is of the int type and the value range is [0, 1]. The value 0 (default) indicates asynchronous execution, and the value 1 indicates synchronous execution. Set the value as required.
      deviceId_	Specifies the processor on which the plugin runs. The value is of the int type and is the same as the value of deviceId at the beginning of the pipeline configuration file, by default. Set the value as required.
      dataSource_	Specifies the source of the data processed by the plugin. The value is of the string type and defaults to auto. You can specify the upstream plugin name as the data source. Set the value as required.

2. Register properties and obtain property values.
   Properties are used to improve the universality of plugins. You can modify the plugin properties in the pipeline configuration file to pass parameters for service adaptation.

   1. Register element properties.

      Rewrite the DefineProperties function and register customized properties as follows:

      1 2 3 4

      std::vector<std::shared_ptr<void>> properties; std::shared_ptr<ElementProperty<string>> outputResizeHeight(new ElementProperty<string> {STRING, "resizeHeight", "resizeHeight", "the height of the resized output image", 640, 6, 4096}); properties.push_back(parentNameSptr); return properties;

      ElementProperty is a struct that describes properties. The content and description are as follows:

      1 2 3 4 5 6 7 8 9 10

      template<class T> struct ElementProperty { PropertyType type; // Property data type. The type can be string, int, uint, float, double, long, ulong. std::string name; // Property name, which is used to obtain the property value. For details, see the section about obtaining the property value. std::string nickName; // Property alias. This field is reserved. std::string desc; // Property description, which describes the function of the property. T defaultValue; // Default value of the property. T min; // Minimum value of the property. Set this field as required. For example, if the data of the string type does not have a minimum value, set this field to NULL. T max; // Maximum value of the property. Set this field as required. For example, if the data of the string type does not have a maximum value, set this field to NULL. };

   2. Obtain the values of the element properties.

      In the Init() function, the property values corresponding to the plugin are transferred through the configParamMap input parameter and can be obtained based on the property name.

      1 2	std::shared_ptr<string> outputResizeHeight = std::static_pointer_cast<string>(configParamMap["resizeHeight"]); std::string parentName = *outputResizeHeight.get();

3. (Optional) Define port properties.

   The input and output ports of the plugin are used to receive input data and send output results, respectively. Multiple input and output ports can be registered for the plugin (see Creating MIMO Plugin Ports).

   The port type can be static or dynamic (see Creating Dynamic Ports). A static port is generated after an element is created, and it must be connected. Otherwise, the stream fails to be created. The number of dynamic ports is variable, and they can be created as required.

   By rewriting the DefineInputPorts and DefineOutputPorts functions, you can define ports. The following uses a single-input static port as an example to describe how to register customized properties:

   • In DefineInputPorts, use GenerateStaticInputPortsInfo() to construct the static input port information MxpiPortInfo. Then, Vision SDK obtains the return value of DefineInputPorts to create a port.
   • The input parameter of the API is a 2-layer vector. The inner layer defines the data format supported by the port. For example, image/jpeg indicates the JPEG image data before decoding. The port format is used to determine whether the element port can be connected. For details about the existing port formats, see Table 3. You can define multiple data formats for a single port, for example, {"metadata/object", "metadata/class"} for an inference plugin.
   • Vectors at the outer layer determine the number of ports. For example, {{"image/jpeg"}, {"metadata/object"}} indicates that two ports are created. The first port is in jpeg format, and the second port is in object format.
     

     The customized string must start with a letter and can contain letters, digits, and underscores (_). You are advised to name ports in the format of level-1 category/level-2 category, for example, image/yuv.

   • Use the same method to define output ports in the DefineOutputPorts function.
     • In the base class:

       1 2 3 4 5 6 7

       MxpiPortInfo MxPluginBase::DefineInputPorts() { MxpiPortInfo defaultInputPortInfo; std::vector<std::vector<std::string>> value = {{"ANY"}}; GenerateStaticInputPortsInfo(value, defaultInputPortInfo); return defaultInputPortInfo; }

     • In the inference plugin:

       1 2 3 4 5 6 7

       MxpiPortInfo MpModelInfer::DefineInputPorts() { MxpiPortInfo inputPortInfo; std::vector<std::vector<std::string>> value = {{"image/yuv"}}; GenerateStaticInputPortsInfo(value, inputPortInfo); return inputPortInfo; }

   The port formats used by the developed plugins are listed in Table 3:

   Table 3 Port formats

   Port	Type
   Original image	image/jpeg
   Video	video/x-h264
   Decoded image	image/yuv image/rgb
   Bounding box	metadata/object
   Image class	metadata/class
   Image attribute	metadata/attribute
   Image feature	metadata/feature-vector
   Tensor	metadata/tensor
   Drawing unit	metadata/osd
   Semantic segmentation pixel	metadata/semanticseg
   Text	metadata/texts
   Key object point	metadata/keypoint
   Result of serialization	result/json-result
   Compatible with any port formats	ANY. Data in any format can be matched. That is, the port can be connected to any port.

   Table 4 Plugin description

   Plugin	Input	Output	Description
   mxpi_imagedecoder	image/jpeg	image/yuv, image/rgb, metadata/object	Image decoding. metadata/object is supported in block division mode.
   mxpi_videodecoder	video/x-h264	image/yuv, image/rgb	Video decoding.
   mxpi_imageresize	image/yuv, image/rgb, metadata/object	image/yuv, image/rgb	Image resizing
   mxpi_tensorinfer	metadata/tensor or image/yuv	metadata/tensor	Tensor inference. Input tensors or images and output tensors.
   mxpi_modelinfer	image/yuv	metadata/object, metadata/class, metadata/attribute, metadata/feature-vector, metadata/tensor	Image inference. The postprocessing result varies depending on the model: object bounding box, attribute, or feature. If postprocessing is not used, the inference result is directly output.
   mxpi_imagecrop	metadata/objectbox	image/yuv, image/rgb	Image cropping
   mxpi_dataserialize	ANY	result/json-result	Serialized into a JSON string.

4. Build and name a plugin framework.
   In the plugin source file, use the following macro to build a plugin framework (CustomPlugin indicates the custom plugin class name):

   1	MX_PLUGIN_GENERATE(CustomPlugin) // Register a custom plugin using macro.

   The plugin name corresponds to the factory parameter in the pipeline configuration file. Register the name in the plugin file CMakeLists.txt as follows (mxpi_customplugin indicates the plugin name):

   1 2	set(PLUGIN_NAME "mxpi_customplugin") add_compile_options("-DPLUGIN_NAME=${PLUGIN_NAME}")

Creating Dynamic Ports

Dynamic ports apply to the scenario where the amount of input or output data is variable, for example, the synchronization plugin mxpi_synchronize. You can configure the amount of input data to be synchronized. The number of dynamic ports is variable. You can create dynamic ports by editing the pipeline configuration file as required.

As shown in Figure 4, the synchronization plugin can create two or three input ports. The corresponding content of the pipeline configuration file is on the right of the figure.

Figure 4 Example of elements with dynamic ports