Video Decoding, Processing, and Display (NVR Scenario)

NVR Video Scenario

This section describes the NVR service processing flow, which includes:

1. Video decoding and display process

   The video decoding and display process involves the VDEC, VPSS, VO, HDMI, and enhanced functions such as Region Management Function, TDE Graphics Drawing, and HiFB for Managing Overlaid Graphics Layers.

   Call hi_mpi_sys_bind to bind the VDEC channel with the VPSS group, and bind the VPSS group with the VO device. In this way, data is automatically transferred from the VDEC to the VPSS, and from the VPSS to the VO device. The VO module can automatically read the output data of the HiFB by using underlying registers.

   If the TDE, HiFB, and VO are not in the same app process, start the VO process first, initialize the media data processing system by calling hi_mpi_sys_init in the TDE and HiFB process, and deinitialize the media data processing system by calling hi_mpi_sys_exit before exiting the process.

2. Intelligent analysis process of video decoding

   The intelligent analysis process of video decoding involves the VDEC, VPSS, intelligent analysis module for model inference, and enhanced functions such as the Region module. For details about the API call sequence of the region management function, see Region Management Function.

Video Decoding and Display Process

The video decoding and display process involves the VDEC, VPSS, VO, HDMI, and enhanced functions such as Region Management Function, TDE Graphics Drawing, and HiFB for Managing Overlaid Graphics Layers.

The API call sequence for starting and running the NVR video decoding and display service is as follows:

1. Call hi_mpi_sys_init to initialize the media system.
2. Create VDEC channels as required and instruct the decoder to start receiving streams.
   1. Call hi_mpi_vdec_create_chn to create a channel.
   2. Call hi_mpi_vdec_get_chn_param to obtain channel attributes, set channel attributes as required, and then call hi_mpi_vdec_set_chn_param to set channel parameters.
   3. Before decoding, call hi_mpi_vdec_start_recv_stream to instruct the decoder to start receiving streams.
3. Create multiple VPSS groups as required and enable them.
   1. Call hi_mpi_vpss_create_grp to create VPSS groups.
   2. Call hi_mpi_vpss_start_grp to enable the VPSS groups.
   3. Call hi_mpi_vpss_set_chn_attr to set the channel attributes. When working with the VO module is needed, set the channel attribute to the auto mode.
   4. Call hi_mpi_vpss_enable_chn to enable the VPSS channel.
4. Call hi_mpi_sys_bind to bind the VDEC channel to a VPSS group. The output data decoded by the VDEC is directly sent to the corresponding VPSS group for further processing.
   • For example, a single monitor outputs four channels of videos. The following table lists the binding information.

     VDEC Channel	VPSS Group	VO Channel	VO Video Layer	VO device ID	HDMI ID
     1	1	1	Video layer VHD0	0	HDMI0
     2	2	2
     3	3	3
     4	4	4

   • For example, two monitors each outputs four channels of videos. The following table lists the binding information.

     VDEC Channel	VPSS Group	VO Channel	VO Video Layer	VO Device ID	HDMI ID
     1	1	1	Video layer VHD0	0	HDMI0
     2	2	2
     3	3	3
     4	4	4
     5	5	5	Video layer VHD1	1	HDMI1
     6	6	6
     7	7	7
     8	8	8

5. Enable the VO device and VO channel (create multiple VO channels as required).
   1. Call hi_mpi_vo_set_pub_attr to set the attributes of the display device, and call hi_mpi_vo_enable to enable the display device.
   2. Call hi_mpi_vo_set_video_layer_attr to set the attributes of the display video layer, and call hi_mpi_vo_enable_video_layer to enable the display video layer.
   3. Call hi_mpi_vo_set_chn_attr to set the attributes of the display channel, and call hi_mpi_vo_enable_chn to enable the display channel.
6. Initialize HDMI peripherals.
   1. Call hi_mpi_hdmi_init to initialize the HDMI device, and call hi_mpi_hdmi_open to enable the HDMI.
   2. Call hi_mpi_hdmi_set_attr to configure the HDMI attributes.
   3. Call hi_mpi_hdmi_start to start the HDMI peripherals to display videos.
7. Call hi_mpi_sys_bind to bind a VPSS group to a VO channel. The output data processed by the VPSS is directly sent to the corresponding VO channel for further processing.

   For details about the binding information, see 4.

8. Call hi_mpi_vdec_send_stream repeatedly to send each frame of decoded streams.

   Note:

   1. After a VDEC channel is bound to a VPSS group (see 4), you can set vdec_pic_info to NULL when call hi_mpi_vdec_send_stream to send streams. The decoding result data is directly sent to the corresponding VPSS group for further processing. You cannot obtain the decoding result data by calling hi_mpi_vdec_get_frame.
   2. Video decoding supports the playback mode and the preview mode. You can call hi_mpi_vdec_get_display_mode and hi_mpi_vdec_set_display_mode to query and set the mode. Playback control is supported when the playback mode is required for recording playback.

The API call sequence for releasing resources of the NVR video decoding and display service is as follows:

1. Call hi_mpi_sys_unbind to unbind the VO channel from the VPSS group.
2. Call hi_mpi_sys_unbind to unbind the VPSS group from the VDEC channel.
3. Release HDMI peripheral resources.
   1. Call hi_mpi_hdmi_stop to stop the HDMI.
   2. Call hi_mpi_hdmi_close to disable the HDMI.
   3. Call hi_mpi_hdmi_deinit to deinitialize the HDMI device.
4. Release the VO device and channel resources.
   1. Call hi_mpi_vo_disable_chn to disable a channel.
   2. Call hi_mpi_vo_disable_video_layer to disable the video layer.
   3. Call hi_mpi_vo_disable to disable the display device.
5. Destroy the VPSS group.
   1. Call hi_mpi_vpss_disable_chn to disable the VPSS channel.
   2. Call hi_mpi_vpss_stop_grp to stop the VPSS group.
   3. Call hi_mpi_vpss_destroy_grp to destroy the VPSS group.
6. Destroy the VDEC channel.
   1. Call hi_mpi_vdec_stop_recv_stream to instruct the decoder to stop receiving streams.
   2. Call hi_mpi_vdec_destroy_chn to destroy the channel.
7. Call hi_mpi_sys_exit to deinitialize the media data processing system.

Intelligent Analysis Process of Video Decoding

The intelligent analysis process of video decoding involves the VDEC, VPSS, intelligent analysis module for model inference, and enhanced functions such as the Region module. For details about the API call sequence of the region management function, see Region Management Function.

The process of starting and running the video decoding intelligent analysis service is as follows:

1. Call hi_mpi_sys_init to initialize the media system.
2. Create VDEC channels as required and instruct the decoder to start receiving streams.
   1. Call hi_mpi_vdec_create_chn to create a channel.
   2. Call hi_mpi_vdec_get_chn_param to obtain channel attributes, set channel attributes as required, and then call hi_mpi_vdec_set_chn_param to set channel parameters.
   3. Before decoding, call hi_mpi_vdec_start_recv_stream to instruct the decoder to start receiving streams.
3. Create multiple VPSS groups as required and enable them.
   1. Call hi_mpi_vpss_create_grp to create VPSS groups.
   2. Call hi_mpi_vpss_start_grp to enable the VPSS groups.
   3. Call hi_mpi_vpss_set_chn_attr to set the channel attribute, which must be set to the user mode.
   4. Call hi_mpi_vpss_enable_chn to enable the VPSS channel.
4. Call hi_mpi_sys_bind to bind the VDEC channel to a VPSS group. The output data decoded by the VDEC is directly sent to the corresponding VPSS group for further processing.

   For details about the binding information, see 4.

5. Call hi_mpi_vdec_send_stream repeatedly to send each frame of decoded streams.

   Note:

   After a VDEC channel is bound to a VPSS group (see 4), you can set vdec_pic_info to NULL when call hi_mpi_vdec_send_stream to send streams. The decoding result data is directly sent to the corresponding VPSS group for further processing. You cannot obtain the decoding result data by calling hi_mpi_vdec_get_frame.

6. Call hi_mpi_vpss_get_chn_frame to obtain the image data processed by the VPSS and send the data for model inference (for details, see Model Management). After the inference is complete, call hi_mpi_vpss_release_chn_frame to release a frame of channel image.

   Note:

   When there are multiple channels, you can obtain the handle to a specified channel by calling hi_mpi_vpss_get_chn_fd, and wait for the processing result by calling the epoll interface (for details, see hi_mpi_sys_create_epoll). After the VPSS completes processing, the epoll is automatically woken up. In this case, you can obtain the processed image data by calling hi_mpi_vpss_get_chn_frame.

   If the size and format of the image processed by the VPSS do not meet the requirements before being sent for inference, the image needs to be further processed by the VPC and AIPP functions of the DVPP. For details, see the descriptions of the VPC and AIPP in Overview.

Region Management Function

The overlaid on screen displays (OSDs) and color blocks in a video are called regions. The Region module is used to manage region resources in a unified manner, and display specific information (such as the channel ID and timestamp) on a video, or fill color blocks in a video for covering.

The Region module must be used together with the VPSS module, and the VPSS group and VPSS associated with the Region module must have been created. The API call sequence is as follows:

1. Initialize.
   1. Call hi_mpi_rgn_create to create a region.
   2. Call hi_mpi_rgn_attach_to_chn to overlay the region on the VPSS channel.
2. Modify region information.
   • Set the channel display attributes of a region.
     1. Call hi_mpi_rgn_get_display_attr to obtain the current channel display attributes of the region.
     2. Call hi_mpi_rgn_set_display_attr to set the channel display attributes of the region.
   • Set the canvas information of the region.
     1. Call hi_mpi_rgn_get_canvas_info to obtain the canvas information of the current region.
     2. Call hi_mpi_rgn_update_canvas to update the display canvas information.
3. Destroy allocations.
   1. Call hi_mpi_rgn_detach_from_chn to delete a specified region from the VPSS channel.
   2. Call hi_mpi_rgn_destroy to destroy the region.

TDE Graphics Drawing

The TDE is a two-dimensional graphics acceleration engine. It uses hardware to provide fast graphics drawing functions for the On Screen Display (OSD) and Graphics User Interface (GUI). The functions include quick copy, quick color filling, and pattern filling. (Currently, only alpha blending is supported.)

1. Start the TDE device by calling hi_tde_open.
2. Create a TDE job by calling hi_tde_begin_job.
3. Call the command execution APIs, such as hi_tde_quick_copy, hi_tde_quick_fill, and hi_tde_pattern_fill.

   Before calling a command execution API, call int ioctl (int fd, FBIOGET_FSCREENINFO, fb_fix_screeninfo *fix) provided by the HiFB to obtain the user mode address of the display buffer as the memory address of the target bitmap. After the TDE job is complete, the data of the target bitmap is stored in the memory address as the input data of the HiFB.

4. If a command fails to be added, call hi_tde_cancel_job to cancel the job. If a command is successfully added, call hi_tde_end_job to submit the job.
5. Wait until the TDE job is complete.

   Currently, hi_tde_wait_for_done is provided to wait for the completion of a specified TDE job, and hi_tde_wait_all_done is provided to wait for the completion of all jobs on the current TDE device.

HiFB for Managing Overlaid Graphics Layers

The HiFB is used to manage overlaid graphics layers. It not only provides the basic functions of Linux framebuffer, but also provides extended functions such as modifying the display start position of a graphics layer and inter-layer alpha.

1. Open the FB device by calling the open function. Device files fb0–fb4 correspond to graphics layers G0–G4. G0 and G1 are HD graphics layers, G2 is the cursor graphics layer, and G3 and G4 are SD graphics layers.

   For details about the FB device, VO device, supported color formats, and resolution of each graphics layer, see Table 1.

2. Pass the command code FBIOGET_VSCREENINFO by calling ioctl to obtain the basic attributes of the screen.
3. Pass the command code FBIOPUT_VSCREENINFO by calling ioctl to set the basic attributes of the screen.
4. Pass the command code FBIOGET_FSCREENINFO by calling ioctl to obtain the user mode address of the display buffer.
5. Map the user-mode virtual address by calling mmap.
6. The system calls ioctl to pass function-related command codes to set function attributes.
7. Unmap the user-mode virtual address by calling munmap.
8. Close the FB device by calling close.

   If the HiFB works with the VO in the same process, you must disable the FB device after disabling the VO by calling hi_mpi_vo_disable.

   If the HiFB and VO are not in the same process, stop the VO app process.