Video Data Obtaining

Overall Process

The API call sequence is as follows:

1. Invoke the hi_mpi_sys_init interface to initialize the media data processing system.
2. Use the MIPI RX ioctl command word to initialize the MIPI/sensor hardware interconnection information. For details, see Initializing the MIPI/Sensor Hardware Interconnection Information.
3. Call the VI functional API to initialize the VI module. For details, see Initializing the VI Module.
4. Call the ISP system control API to initialize and run the ISP module. For details, see Initializing and Running the ISP Module.
5. Call the VI functional API to obtain the processed image data. For details, see Obtaining Processed Image Data.
6. Call the ISP functional API to free ISP module resources. For details, see Releasing the Resources of the ISP Module.
7. Call the VI functional API to free VI module resources. For details, see Releasing VI Resources.
8. Use the MIPI RX ioctl command word to exit the MIPI/sensor hardware. For details, see Exiting the MIPI/Sensor Hardware.
9. Call the hi_mpi_sys_exit interface to release the resources of the media data processing system.

Initializing the MIPI/Sensor Hardware Interconnection Information

1. Run the HI_MIPI_SET_HS_MODE command to set the mode.
2. Run the HI_MIPI_ENABLE_MIPI_CLOCK command to enable the MIPI clock.
3. Run the HI_MIPI_RESET_MIPI command to reset the MIPI connected to the sensor.
4. (Optional) Run the HI_MIPI_CONFIG_SENSOR_CLOCK command to configure the sensor clock.
5. (Optional) Run the HI_MIPI_ENABLE_SENSOR_CLOCK command to enable the sensor clock.
6. Run the HI_MIPI_RESET_SENSOR command to reset the sensor.
7. Use the HI_MIPI_SET_DEV_ATTR command word to configure the MIPI RX/device attributes.
8. Run the HI_MIPI_UNRESET_MIPI command to deassert the reset on the MIPI.
9. Run the HI_MIPI_UNRESET_SENSOR command word to deassert the reset on the sensor.

Initializing the VI Module

The process of initializing the VI module varies according to the data source, data format, and mode.

1. Transfer data from the sensor. To obtain YUV data, process the data through the VI channel in linear mode.

   

   Process description:

   1. Set the attributes of the VI device and enable the VI device by calling hi_mpi_vi_set_dev_attr and hi_mpi_vi_enable_dev in sequence.
   2. Bind the device to the pipe by calling hi_mpi_vi_set_dev_bind_pipe.
   3. Create and start a VI pipe by calling hi_mpi_vi_create_pipe and hi_mpi_vi_start_pipe in sequence. The hi_vi_pipe_attr.depth queue depth must be properly set. A larger queue depth indicates better anti-jitter performance. It is recommended that you set the queue depth to 3 or a larger value.
   4. Set the attributes of the VI channel and enable the VI channel by calling hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn in sequence. The hi_vi_chn_attr.depth queue depth needs to be properly set. The queue depth needs to consider the memory reserved for VI internal processing and the image service processing duration (the interval between the time when the hi_mpi_vi_get_chn_frame interface is called to obtain image resources and the time when the hi_mpi_vi_release_chn_frame interface is called to return image resources). Set the queue depth properly.
2. Transfer data from the sensor. To obtain YUV data, use the VI channel in WDR mode.

   

   Compared with the common linear mode, in WDR mode, the sensor module generates two frames of image data at the same time in long and short exposure mode. The VI module needs to create two pipes and bind them to a VI device to receive and process the corresponding long and short exposure frames. Then, the image data after the combination of long and short exposure is output in the channel corresponding to the primary pipe. Therefore, the differences exist in the calling process:

   1. You need to call hi_mpi_vi_set_dev_bind_pipe to bind the image data of the same sensor to two pipes. As shown in the following figure, DEV0 is bound to pipe 0 and pipe 1.
   2. You need to create and start two pipes by calling hi_mpi_vi_create_pipe and hi_mpi_vi_start_pipe. As shown in the preceding figure, pipe 0 and pipe 1 are created. Pipe 0 functions as the primary pipe to receive and process short exposure frames, and pipe 1 functions as the secondary pipe to receive and process long exposure frames.

      In WDR mode, pipe 0 and pipe 1 form a group and pipe 0 is the primary pipe. Pipe 1 and pipe 2 form a group, and pipe 1 is the primary pipe.

   3. Only the channels on the primary pipe need to be enabled. The channels on the secondary pipe do not need to be enabled to save resources.
3. To obtain RAW data, use a VI pipe.

   

   Process description:

   1. Set the attributes of the VI device and enable the VI device by calling hi_mpi_vi_set_dev_attr and hi_mpi_vi_enable_dev.
   2. Bind the device to the pipe by calling hi_mpi_vi_set_dev_bind_pipe.
   3. Create a pipe by calling hi_mpi_vi_create_pipe.
      1. If you only need to obtain raw images without VI processing and conversion, perform the following operations when creating a pipe in hi_mpi_vi_create_pipe:
         1. Set pipe_bypass_mode to HI_VI_PIPE_BYPASS_BE so that the ISP_BE processing is not performed.
         2. Set hi_vi_pipe_attr.depth to hi_vi_dump_attr.depth. Only resources for dumping the required image queue are applied for.
         3. You need to call hi_mpi_vi_set_chn_attr instead of hi_mpi_vi_enable_chn to enable the VI channel.
      2. If you need to send images to the VI module for processing and conversion after obtaining RAW images:
         1. When creating a pipe using hi_mpi_vi_create_pipe, you need to properly set hi_vi_pipe_attr.depth. The value of hi_vi_pipe_attr.depth needs to be reserved for the queue depth required for internal processing of the VI pipe based on the value of hi_vi_dump_attr.depth. Generally, the value of hi_vi_pipe_attr.depth is hi_vi_dump_attr.depth + 3.
         2. Call the hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn APIs to enable the VI channel, call the hi_mpi_vi_get_chn_frame API to obtain and process the image result data processed by the VI channel, and then call the hi_mpi_vi_release_chn_frame API to release the memory resources of the corresponding image.
   4. Invoke the hi_mpi_vi_set_pipe_frame_dump_attr interface to set the depth of the image queue to be reserved for image capture.
   5. Start the pipe by calling hi_mpi_vi_start_pipe.
4. The user specifies the raw image data, and the VI pipe imports and processes the data to obtain the YUV image.

   

   In the image injection scenario, the image data source is no longer an external camera. However, the current version does not support virtual pipes, and only physical pipes can be used for image injection. Therefore, even if the data is not input from the sensor, you still need to set the corresponding device and call the hi_mpi_vi_set_dev_bind_pipe interface to bind the device to the pipe.

   1. Create a pipe by calling hi_mpi_vi_create_pipe.
   2. Call the hi_mpi_vi_set_pipe_frame_source API to set the image data source of the pipe to VI_PIPE_FRAME_SOURCE_USER.
   3. Start the pipe by calling hi_mpi_vi_start_pipe.
   4. Set the attributes of the VI channel and enable the VI channel by calling hi_mpi_vi_set_chn_attr and hi_mpi_vi_enable_chn.
   5. Start to cyclically send the image data specified by the user.
      1. Call hi_mpi_vi_pipe_get_buffer to obtain idle image data. The maximum amount of available memory that can be obtained is determined by the hi_vi_pipe_attr.depth attribute delivered by the hi_mpi_vi_create_pipe API.
      2. After the available memory resources are obtained, write the image data to be dumped to the returned memory address. The memory address is the frame_info.v_frame.virt_addr[0] returned by the hi_mpi_vi_pipe_get_buffer API. Then call the hi_mpi_vi_send_pipe_raw API to send the raw image data.
      3. After the hi_mpi_vi_send_pipe_raw data is successfully sent, the hi_mpi_vi_pipe_release_buffer interface needs to be invoked in a timely manner to release memory resources.
      4. The following figure shows the process of sending Bayer images. To send YUV data, you need to set pixel_format to YUV, set isp_bypass to true, and change hi_mpi_vi_send_pipe_raw to hi_mpi_vi_send_pipe_yuv when creating a pipe by calling hi_mpi_vi_create_pipe. The current version supports the following YUV image formats: HI_PIXEL_FORMAT_YVU_SEMIPLANAR_422, HI_PIXEL_FORMAT_YVU_SEMIPLANAR_420, and HI_PIXEL_FORMAT_YUV_400.

Initializing and Running the ISP Module

1. Call the hi_mpi_isp_sensor_reg_callback interface to register the general algorithm of the sensor driver.
2. (Optional) Call the hi_mpi_ae_sensor_reg_callback and hi_mpi_awb_sensor_reg_callback interfaces to register the AE and AWB algorithms of the built-in sensor driver.

   You can register a custom algorithm as required.

3. (Optional) Call the hi_mpi_ae_register and hi_mpi_awb_register APIs to initialize the built-in 2A algorithm.

   You can register a custom algorithm as required.

4. Initialize the internal resources of the ISP by calling hi_mpi_isp_mem_init.
5. Initialize the algorithm module by calling the hi_mpi_isp_set_pub_attr interface.
6. Initialize the ISP firmware by calling hi_mpi_isp_init.
7. Start an independent thread and call the hi_mpi_isp_run interface to run the main scheduling service process of the ISP algorithm.

Obtaining Processed Image Data

• Obtaining YUV data

  

  After images are processed by VI, obtain the images from the VI channel for processing. The typical API call sequence is as follows:

  1. (Optional) Wait for the image processing completion event by using the system file descriptor + select/epoll waiting mechanism. You can obtain the system file descriptor of a specified channel by using the hi_mpi_vi_get_chn_fd interface. After a frame of image data is obtained and processed, the select/epoll read waiting request of the system is woken up.
  2. Call the hi_mpi_vi_get_chn_frame API to obtain the processed image data. In this case, the memory resources corresponding to the image data are automatically occupied by the user. After the image data is processed, the user needs to call the hi_mpi_vi_release_chn_frame interface to release the memory resources of the corresponding image.
  3. If you want to release the image data obtained by calling the hi_mpi_vi_get_chn_frame API to other processes, you can obtain the acltdtBuf handle based on the returned hi_video_frame.user_data[0], and then use the shared buffer management API (such as acltdtCopyBufRef) and shared queue management API. (for example, acltdtEnqueue) Release the object to other processes.
• Obtaining RAW data

  

  After images are processed by VI, obtain the images from the VI pipe for processing. The typical API call sequence is as follows:

  1. (Optional) Wait for the image processing completion event by using the system file descriptor + select/epoll waiting mechanism. You can obtain the system file descriptor of a specified channel by using the hi_mpi_vi_get_pipe_fd interface. After the background obtains and processes a frame of image data, the select/epoll read waiting request of the system is woken up.
  2. Call the hi_mpi_vi_get_pipe_frame API to obtain the processed image data. In this case, the memory resources corresponding to the image data are automatically occupied by the user. After the image data is processed, the user needs to call the hi_mpi_vi_release_pipe_frame interface to release the memory resources of the corresponding image.
  3. If you want to release the image data obtained by calling the hi_mpi_vi_get_pipe_frame API to other processes, you can obtain the acltdtBuf handle based on the returned hi_video_frame.user_data[0], and then use the shared buffer management API (such as acltdtCopyBufRef) and shared queue management API. (for example, acltdtEnqueue) Release the object to other processes.

Releasing the Resources of the ISP Module

1. Deinitialize the ISP firmware by calling hi_mpi_isp_exit.
2. Call the hi_mpi_ae_unregister and hi_mpi_awb_unregister interfaces to deinitialize the 2A algorithm.
3. Call the hi_mpi_ae_sensor_unreg_callback and hi_mpi_awb_sensor_unreg_callback interfaces to deregister the AE and AWB algorithms of the sensor driver.
4. Call the hi_mpi_isp_sensor_unreg_callback interface to deregister the general algorithm of the sensor driver.

Releasing VI Resources

1. Disable the VI channel by calling hi_mpi_vi_disable_chn.
2. Stop and destroy the VI pipe by calling hi_mpi_vi_stop_pipe and hi_mpi_vi_destroy_pipe in sequence.
3. Stop the VI device by calling hi_mpi_vi_disable_dev.

Exiting the MIPI/Sensor Hardware

1. Run the HI_MIPI_RESET_SENSOR command to reset the sensor.
2. Run the HI_MIPI_DISABLE_SENSOR_CLOCK command to disable the clock connected to the sensor.
3. Run the HI_MIPI_RESET_MIPI command to reset the MIPI connected to the sensor.
4. Use HI_MIPI_DISABLE_MIPI_CLOCK to disable the MIPI.