Video Decoding, Processing, and Display (NVR Scenario)

NVR Video Scenario

This section describes the NVR service processing flow, which includes:

1. Video decoding and display process

   The video decoding and display process involves the VDEC, VPSS, VO, HDMI, and enhanced functions such as Region Management Function, TDE graphics drawing, and HiFB graphics layer management.

   In this process, the VDEC channel is bound to the VPSS group and the VPSS group is bound to the VO device by calling the hi_mpi_sys_bind interface. In this way, data is automatically transferred from the VDEC to the VPSS and from the VPSS to the VO. The VO module can automatically read the output data of the HiFB by using underlying registers.

   If the TDE, HiFB, and VO are not in the same application process, ensure that the process where the VO is located is started first. In the process where the TDE and HiFB are located, the media data processing system needs to be initialized by calling the hi_mpi_sys_init interface, and the media data processing system needs to be deinitialized by calling the hi_mpi_sys_exit interface before the process exits.

2. Intelligent analysis process of video decoding

   The intelligent analysis process of video decoding involves the VDEC, VPSS, intelligent analysis module for model inference, and enhanced functions such as the Region module. For details about the API call sequence of the region management function, see Region Management Function.

Video Decoding and Display Process

The video decoding and display process involves the VDEC, VPSS, VO, HDMI, and enhanced functions such as Region Management Function, TDE Graphics Drawing, and HiFB for Managing Overlaid Graphics Layers.

The API call sequence for starting and running the NVR video decoding and display service is as follows:

1. Call the hi_mpi_sys_init API to initialize the media system.
2. Create VDEC channels as required and instruct the decoder to start receiving streams.
   1. Call the hi_mpi_vdec_create_chn API to create a channel.
   2. Call the hi_mpi_vdec_get_chn_param API to obtain the channel attributes, set the channel attributes as required, and then call the hi_mpi_vdec_set_chn_param API to set channel parameters.
   3. Before decoding, invoke the hi_mpi_vdec_start_recv_stream interface to instruct the decoder to start receiving streams.
3. Create multiple VPSS groups as required and enable them.
   1. Create a VPSS group by calling hi_mpi_vpss_create_grp.
   2. Enable a VPSS group by calling hi_mpi_vpss_start_grp.
   3. Set the channel attributes by calling hi_mpi_vpss_set_chn_attr. When the VOU works with the VO module, the channel attribute must be set to auto mode.
   4. Start the VPSS channel by calling hi_mpi_vpss_enable_chn.
4. Call the hi_mpi_sys_bind API to bind a VDEC channel to a VPSS group. The output data after VDEC is directly sent to the corresponding VPSS group for processing.
   • For example, a single monitor outputs four channels of videos. The following table lists the binding information.

     VDEC Channel	VPSS Group	VO Channel	VO Video Layer	VO device ID	HDMI ID
     1	1	1	Video layer VHD0	0	HDMI0
     2	2	2
     3	3	3
     4	4	4

   • For example, two monitors each outputs four channels of videos. The following table lists the binding information.

     VDEC Channel	VPSS Group	VO Channel	VO Video Layer	VO Device ID	HDMI ID
     1	1	1	Video layer VHD0	0	HDMI0
     2	2	2
     3	3	3
     4	4	4
     5	5	5	Video layer VHD1	1	HDMI1
     6	6	6
     7	7	7
     8	8	8

5. Enable the VO device and VO channel (create multiple VO channels as required).
   1. Call hi_mpi_vo_set_pub_attr to configure the display device attributes, and call hi_mpi_vo_enable to enable the display device.
   2. Set the attributes of the display video layer by calling hi_mpi_vo_set_video_layer_attr, and enable the display video layer by calling hi_mpi_vo_enable_video_layer.
   3. Call hi_mpi_vo_set_chn_attr to configure the attributes of the display channel, and call hi_mpi_vo_enable_chn to enable the display channel.
6. Initialize HDMI peripherals.
   1. Initialize the HDMI device by calling hi_mpi_hdmi_init, and enable the HDMI by calling hi_mpi_hdmi_open.
   2. Configure the HDMI attributes by calling hi_mpi_hdmi_set_attr.
   3. Invoke the hi_mpi_hdmi_start interface to start the HDMI peripheral to display videos.
7. Call the hi_mpi_sys_bind API to bind a VPSS group to a VO channel. The output data processed by the VPSS is directly sent to the corresponding VO channel for further processing.

   For details about the binding information, see 4.

8. Call the hi_mpi_vdec_send_stream API cyclically to send the decoded streams of each frame.

   Note:

   1. After a VDEC channel is bound to a VPSS group (see 4), you can set vdec_pic_info to NULL when call hi_mpi_vdec_send_stream to send streams. The decoding result data is directly sent to the corresponding VPSS group for further processing. You cannot obtain the decoding result data by calling hi_mpi_vdec_get_frame.
   2. Video decoding supports two modes: playback mode and preview mode, which can be queried and set by calling hi_mpi_vdec_get_display_mode and hi_mpi_vdec_set_display_mode. The playback mode is required for recording playback. In this case, playback control is supported.

The API call sequence for releasing resources of the NVR video decoding and display service is as follows:

1. Call the hi_mpi_sys_unbind API to unbind a VO channel from a VPSS group.
2. Call the hi_mpi_sys_unbind API to unbind the VPSS group from the VDEC channel.
3. Release HDMI peripheral resources.
   1. Stop the HDMI by calling hi_mpi_hdmi_stop.
   2. Disable the HDMI by calling hi_mpi_hdmi_close.
   3. Deinitialize the HDMI device by calling hi_mpi_hdmi_deinit.
4. Release the VO device and channel resources.
   1. Call hi_mpi_vo_disable_chn to disable the channel.
   2. Disable the video layer by calling hi_mpi_vo_disable_video_layer.
   3. Disable the display device by calling hi_mpi_vo_disable.
5. Destroy the VPSS group.
   1. Disable the VPSS channel by calling hi_mpi_vpss_disable_chn.
   2. Stop the VPSS group by calling hi_mpi_vpss_stop_grp.
   3. Destroy the VPSS group by calling hi_mpi_vpss_destroy_grp.
6. Destroy the VDEC channel.
   1. Invoke the hi_mpi_vdec_stop_recv_stream interface to instruct the decoder to stop receiving streams.
   2. Destroy the channel by calling hi_mpi_vdec_destroy_chn.
7. Call the hi_mpi_sys_exit API to deinitialize the media data processing system.

Intelligent Analysis Process of Video Decoding

The intelligent analysis process of video decoding involves the VDEC, VPSS, intelligent analysis module for model inference, and enhanced functions such as the Region module. For details about the API call sequence of the region management function, see Region Management Function.

The process of starting and running the video decoding intelligent analysis service is as follows:

1. Call the hi_mpi_sys_init API to initialize the media system.
2. Create VDEC channels as required and instruct the decoder to start receiving streams.
   1. Call the hi_mpi_vdec_create_chn API to create a channel.
   2. Call the hi_mpi_vdec_get_chn_param API to obtain the channel attributes, set the channel attributes as required, and then call the hi_mpi_vdec_set_chn_param API to set channel parameters.
   3. Before decoding, invoke the hi_mpi_vdec_start_recv_stream interface to instruct the decoder to start receiving streams.
3. Create multiple VPSS groups as required and enable them.
   1. Create a VPSS group by calling hi_mpi_vpss_create_grp.
   2. Enable a VPSS group by calling hi_mpi_vpss_start_grp.
   3. Set channel attributes by calling hi_mpi_vpss_set_chn_attr. The channel mode must be set to user mode.
   4. Start the VPSS channel by calling hi_mpi_vpss_enable_chn.
4. Call the hi_mpi_sys_bind API to bind a VDEC channel to a VPSS group. The output data after VDEC is directly sent to the corresponding VPSS group for processing.

   For details about the binding information, see 4.

5. Call the hi_mpi_vdec_send_stream API cyclically to send the decoded streams of each frame.

   Note:

   After a VDEC channel is bound to a VPSS group (see 4), you can set vdec_pic_info to NULL when call hi_mpi_vdec_send_stream to send streams. The decoding result data is directly sent to the corresponding VPSS group for further processing. You cannot obtain the decoding result data by calling hi_mpi_vdec_get_frame.

6. Call the hi_mpi_vpss_get_chn_frame API to obtain the image data processed by the VPSS and send the data to the model for inference. After the inference is complete, call the hi_mpi_vpss_release_chn_frame API to release a frame of channel image.

   Note:

   In multi-channel mode, you can call the hi_mpi_vpss_get_chn_fd interface to obtain the handle of a specified VPSS channel, and call the epoll interface (for details, see the hi_mpi_sys_create_epoll interface) to wait for the VPSS processing result. After the VPSS processing is complete, the epoll wait is automatically woken up. In this case, you can call the hi_mpi_vpss_get_chn_frame interface to obtain the image data processed by the VPSS.

   Before the images processed by the VPSS are sent to the model for inference, if the image size and format do not meet the requirements, the images need to be further processed by the VPC and AIPP modules of the DVPP. For details, see the description of VPC and AIPP in Introduction to the Media Data Processing Function.

Region Management Function

The overlaid on screen displays (OSDs) and color blocks in a video are called regions. The Region module is used to manage region resources in a unified manner, and display specific information (such as the channel ID and timestamp) on a video, or fill color blocks in a video for covering.

The Region module must be used together with the VPSS module, and the VPSS group and VPSS associated with the Region module must have been created. The API call sequence is as follows:

1. Initialize.
   1. Call hi_mpi_rgn_create to create a region.
   2. Call the hi_mpi_rgn_attach_to_chn interface to overlay the region on the VPSS channel.
2. Modify region information.
   • Set the channel display attributes of a region.
     1. Obtain the current channel display attributes of a region by calling hi_mpi_rgn_get_display_attr.
     2. Set the channel display attributes of a region by calling hi_mpi_rgn_set_display_attr.
   • Set the canvas information of the region.
     1. Invoke the hi_mpi_rgn_get_canvas_info interface to obtain the canvas information of the current region.
     2. Invoke the hi_mpi_rgn_update_canvas interface to update the canvas information.
3. Destroy allocations.
   1. Delete a specified region from a VPSS channel by calling hi_mpi_rgn_detach_from_chn.
   2. Call the hi_mpi_rgn_destroy interface to destroy the region.

TDE Graphics Drawing

The TDE is a two-dimensional graphics acceleration engine. It uses hardware to provide fast graphics drawing functions for the On Screen Display (OSD) and Graphics User Interface (GUI). The functions include quick copy, quick color filling, and pattern filling. (Currently, only alpha blending is supported.)

HiFB for Managing Overlaid Graphics Layers

The HiFB is used to manage overlaid graphics layers. It not only provides the basic functions of Linux framebuffer, but also provides extended functions such as modifying the display start position of a graphics layer and inter-layer alpha.