Documentation

Video Data Obtaining

Video data can be obtained through the cooperation between functional modules such as the ISP, MIPI RX, and VI. This section describes the overall API call sequence, API call sequence regarding each module, and precautions for obtaining video data.

Currently, the following functional modules are required to obtain video data:

  • ISP system control

    The system control function is used to register the 3A algorithm, register the sensor driver, initialize the ISP firmware, run the ISP firmware, exit the ISP firmware, and configure the image signal processing (ISP) attributes.

  • MIPI RX ioctl command words

    MIPI RX is a collection unit that supports multiple differential video input interfaces. It receives data from the MIPI, LVDS, sub-LVDS, and HiSPI interfaces through the combo PHY. MIPI RX supports data transmission at multiple speeds and resolutions by configuring different function modes and supports multiple external input devices.

  • Video Input (VI)

    The VI module captures video images, performs operations such as cropping, color optimization, brightness optimization, and noise removal on the images, and outputs YUV or RAW images.

Overall Process

The API call sequence is as follows:

  1. Call hi_mpi_sys_init to initialize the media data processing system.
  2. Use the MIPI RX ioctl command word to initialize the MIPI/sensor hardware interconnection information. For details, see Initializing the MIPI/Sensor Hardware Interconnection Information.
  3. Call the VI functional API to initialize the VI module. For details, see Initializing the VI Module.
  4. Call the ISP system control API to initialize and run the ISP module. For details, see Initializing and Running the ISP Module.
  5. Call the VI functional API to obtain the processed image data. For details, see Obtaining Processed Image Data.
  6. Call the ISP functional API to free ISP module resources. For details, see Releasing the Resources of the ISP Module.
  7. Call the VI functional API to free VI module resources. For details, see Releasing VI Resources.
  8. Use the MIPI RX ioctl command word to exit the MIPI/sensor hardware. For details, see Exiting the MIPI/Sensor Hardware.
  9. Call hi_mpi_sys_exit to free the resources of the media data processing system.

Initializing the MIPI/Sensor Hardware Interconnection Information

  1. Run the HI_MIPI_SET_HS_MODE command word to set the mode.
  2. Run the HI_MIPI_ENABLE_MIPI_CLOCK command word to enable the MIPI clock.
  3. Run the HI_MIPI_RESET_MIPI command word to reset the MIPI connected to the sensor.
  4. (Optional) Run the HI_MIPI_CONFIG_SENSOR_CLOCK command word to configure the sensor clock.
  5. (Optional) Run the HI_MIPI_ENABLE_SENSOR_CLOCK command word to enable the sensor clock.
  6. Run the HI_MIPI_RESET_SENSOR command word to reset the sensor.
  7. Run the HI_MIPI_SET_DEV_ATTR command word to configure the attributes of the MIPI Rx device.
  8. Run the HI_MIPI_UNRESET_MIPI command word to deassert the reset on the MIPI device.
  9. Run the HI_MIPI_UNRESET_SENSOR command word to cancel the sensor resetting.

Initializing the VI Module

The process of initializing the VI module varies according to the data source, data format, and mode.

Solution

API call sequence

Process Description

Obtain input data from a sensor. Especially, obtain YUV input data through a VI channel in linear mode.

  1. Set the attributes of the VI device and enable the VI device by calling hi_mpi_vi_set_dev_attr and hi_mpi_vi_enable_dev in sequence.
  2. Bind the device to the pipe by calling hi_mpi_vi_set_dev_bind_pipe.
  3. Create and start a VI pipe by calling hi_mpi_vi_create_pipe and hi_mpi_vi_start_pipe in sequence. The hi_vi_pipe_attr.depth queue depth must be properly set. A larger queue depth indicates better anti-jitter performance. It is recommended that you set the queue depth to 3 or a larger value.
  4. Set the attributes of the VI channel and enable the VI channel by calling hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn in sequence. The hi_vi_chn_attr.depth queue depth must be properly set. In addition to the buffer reserved for VI internal processing, the user's image service processing duration (interval from the time when the user obtains image resources by calling hi_mpi_vi_get_chn_frame to the time when the user returns image resources by calling hi_mpi_vi_release_chn_frame) must be considered.

Obtain input data from a sensor. Especially, obtain YUV input data through a VI channel in WDR mode.

Compared with the common linear mode, in WDR mode, the sensor module generates two frames of image data at the same time in long and short exposure mode. The VI module needs to create two pipes and bind them to a VI device to receive and process the corresponding long and short exposure frames. Then, the image data after the combination of long and short exposure is output in the channel corresponding to the primary pipe. Therefore, the differences exist in the calling process:

  1. Call hi_mpi_vi_set_dev_bind_pipe to bind the image data of one sensor device to two pipes. As shown in the example figure, device 0 is bound to pipe 0 and pipe 1.
  2. Create and start two pipes by calling hi_mpi_vi_create_pipe and hi_mpi_vi_start_pipe. As shown in the example figure, pipe 0 and pipe 1 are created. Pipe 0 functions as the primary pipe to receive and process short exposure frames, and pipe 1 functions as the secondary pipe to receive and process long exposure frames.
    NOTE:

    In WDR mode, pipe 0 and pipe 1 form a group and pipe 0 is the primary pipe. Pipe 1 and pipe 2 form a group, and pipe 1 is the primary pipe.

  3. Only the channels on the primary pipe need to be enabled. The channels on the secondary pipe do not need to be enabled to save resources.

Obtain raw data through a VI pipe.

  1. Set the attributes of the VI device and enable the VI device by calling hi_mpi_vi_set_dev_attr and hi_mpi_vi_enable_dev.
  2. Bind the device to the pipe by calling hi_mpi_vi_set_dev_bind_pipe.
  3. Create a pipe by calling hi_mpi_vi_create_pipe.
    1. If you only need to obtain RAW images without VI processing and conversion, perform the following operations when creating a pipe in hi_mpi_vi_create_pipe:
      1. Set pipe_bypass_mode to HI_VI_PIPE_BYPASS_BE so that the ISP_BE processing is not performed.
      2. Set hi_vi_pipe_attr.depth to hi_vi_dump_attr.depth. Only resources for dumping the required image queue are applied for.
      3. Call hi_mpi_vi_set_chn_attr, while hi_mpi_vi_enable_chn does not need to be called to enable the VI channel.
    2. If you need to send images to the VI module for processing and conversion after obtaining RAW images:
      1. When a pipe is created by calling hi_mpi_vi_create_pipe, the value of hi_vi_pipe_attr.depth needs to be properly set as some queue depth required for internal processing of the VI pipe needs to be reserved based on hi_vi_dump_attr.depth. Generally, the value is hi_vi_dump_attr.depth + 3.
      2. Enable the VI channel by calling hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn. Obtain and process the images processed by the VI channel by calling hi_mpi_vi_get_chn_frame, and then release the buffer for storing the images by calling hi_mpi_vi_release_chn_frame.
  4. Call hi_mpi_vi_set_pipe_frame_dump_attr to set the depth of the image queue to be reserved for image capturing.
  5. Start the pipe by calling hi_mpi_vi_start_pipe.

Obtain raw data specified by the user through a VI pipe and process the data to obtain a YUV image.

When the user imports an image, the image data source is no longer an external camera device. However, the current version does not support virtual pipes and images can be injected only through physical pipes. Therefore, even if data is not input from the sensor, you must set the corresponding device and call hi_mpi_vi_set_dev_bind_pipe to bind the device to the pipe.

  1. Call hi_mpi_vi_create_pipe to create a pipe.
  2. Set the image data source of the pipe to VI_PIPE_FRAME_SOURCE_USER by calling hi_mpi_vi_set_pipe_frame_source.
  3. Start the pipe by calling hi_mpi_vi_start_pipe.
  4. Set the attributes of the VI channel and enable the VI channel by calling hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn.
  5. Start to cyclically send the image data specified by the user.
    1. Obtain idle image data by calling hi_mpi_vi_pipe_get_buffer. The maximum amount of available buffer that can be obtained is determined by the hi_vi_pipe_attr.depth attribute delivered by hi_mpi_vi_create_pipe.
    2. After the available buffer resources are successfully obtained, write the image data to be imported to the returned buffer address. The buffer address is frame_info.v_frame.virt_addr[0] returned by hi_mpi_vi_pipe_get_buffer. Then call hi_mpi_vi_send_pipe_raw to send the RAW image data.
    3. After hi_mpi_vi_send_pipe_raw successfully sends data, call hi_mpi_vi_pipe_release_buffer in time to release buffer resources.
    4. The preceding figure shows the process of sending Bayer images. To send YUV data, set pixel_format to YUV and isp_bypass to true, and change hi_mpi_vi_send_pipe_raw to hi_mpi_vi_send_pipe_yuv when creating a pipe by calling hi_mpi_vi_create_pipe. The current version supports the following YUV image formats: HI_PIXEL_FORMAT_YVU_SEMIPLANAR_422, HI_PIXEL_FORMAT_YVU_SEMIPLANAR_420, and HI_PIXEL_FORMAT_YUV_400.

Initializing and Running the ISP Module

  1. Call hi_mpi_isp_sensor_reg_callback to register the common algorithm of the sensor driver.
  2. (Optional) Call hi_mpi_ae_sensor_reg_callback and hi_mpi_awb_sensor_reg_callback to register the AE and AWB algorithms (2A algorithms) of the built-in sensor driver.

    You can register a custom algorithm as required.

  3. (Optional) Call hi_mpi_ae_register and hi_mpi_awb_register to initialize the built-in 2A algorithms.

    You can register a custom algorithm as required.

  4. Call hi_mpi_isp_mem_init to initialize the internal resources of the ISP module.
  5. Call hi_mpi_isp_set_pub_attr to initialize the algorithm module.
  6. Call hi_mpi_isp_init to initialize the ISP firmware.
  7. Enable a separate thread and call hi_mpi_isp_run to run the main scheduling service process of the ISP algorithm.

Obtaining Processed Image Data

  • Obtaining YUV data

    After images are processed by VI, obtain the images from the VI channel for processing. The typical API call sequence is as follows:
    1. (Optional) Wait for the image processing completion event by using the system file descriptor and select/epoll wait mechanism. You can obtain the system file descriptor of a specified channel by calling hi_mpi_vi_get_chn_fd. After a frame of image data is obtained and processed, the select/epoll read wait request of the system is woken up.
    2. Obtain the processed image data by calling hi_mpi_vi_get_chn_frame. In this case, the memory corresponding to the image data is automatically occupied. You must free the memory after processing the image data by calling hi_mpi_vi_release_chn_frame.
    3. If you want to deliver the image data obtained by calling hi_mpi_vi_get_chn_frame to other processes, obtain the acltdtBuf handle by using the returned hi_video_frame.user_data[0], and then use the shared buffer management API (such as acltdtCopyBufRef) and the shared queue management API (such as acltdtEnqueue) to deliver objects to other processes.
  • Obtaining RAW data

    After images are processed by VI, obtain the images from the VI pipe for processing. The typical API call sequence is as follows:
    1. (Optional) Wait for the image processing completion event by using the system file descriptor and select/epoll wait mechanism. You can obtain the system file descriptor of a specified channel by calling hi_mpi_vi_get_pipe_fd. After a frame of image data is obtained and processed in the background, the select/epoll read wait request of the system is woken up.
    2. Obtain the processed image data by calling hi_mpi_vi_get_pipe_frame. In this case, the memory corresponding to the image data is automatically occupied. You must free the memory after processing the image data by calling hi_mpi_vi_release_pipe_frame.
    3. If you want to deliver the image data obtained by calling hi_mpi_vi_get_pipe_frame to other processes, obtain the acltdtBuf handle by using the returned hi_video_frame.user_data[0], and then use the shared buffer management API (such as acltdtCopyBufRef) and the shared queue management API (such as acltdtEnqueue) to deliver objects to other processes.

Releasing the Resources of the ISP Module

  1. Call hi_mpi_isp_exit to deinitialize the ISP firmware.
  2. Call hi_mpi_ae_unregister and hi_mpi_awb_unregister to deinitialize the AE and AWB algorithms.
  3. Call hi_mpi_ae_sensor_unreg_callback and hi_mpi_awb_sensor_unreg_callback to deregister the AE and AWB algorithms of the sensor driver.
  4. Call hi_mpi_isp_sensor_unreg_callback to deregister the common algorithms of the sensor driver.

Releasing VI Resources

  1. Disable the VI channel by calling hi_mpi_vi_disable_chn.
  2. Stop and destroy the VI pipe by calling hi_mpi_vi_stop_pipe and hi_mpi_vi_destroy_pipe in sequence.
  3. Stop the VI device by calling hi_mpi_vi_disable_dev.

Exiting the MIPI/Sensor Hardware

  1. Run the HI_MIPI_RESET_SENSOR command word to reset the sensor.
  2. Run the HI_MIPI_DISABLE_SENSOR_CLOCK command word to disable the clock connected to the sensor.
  3. Run the HI_MIPI_RESET_MIPI command word to reset the MIPI connected to the sensor.
  4. Run the HI_MIPI_DISABLE_MIPI_CLOCK command word to disable the MIPI clock.
This site uses cookies to improve your browsing experience. By continuing to browse this site, you agree to our use of cookies.View Details