ISP Firmware Introduction

ISP Firmware Architecture

The image signal processing (ISP) firmware consists of three parts: ISP control unit and ISP base algorithm library, AE/AWB/AF (3A) algorithm library (AF not supported yet), and sensor library. Only the Atlas 200I/500 A2 inference products supports functions in this part.

Figure 1 Architecture of the ISP firmware

Design principles of the ISP firmware:

An independent 3A algorithm library is provided. The ISP control unit schedules the ISP base algorithm library and 3A algorithm library.
The sensor library on the service side registers callback functions with the ISP base algorithm library and 3A algorithm library to support various sensors.
Different sensors register callback functions with the ISP base algorithm library. When the ISP control unit schedules the base algorithm unit and 3A algorithm library, the callback functions are called to obtain initialization parameters and control sensors, for example, adjusting the exposure time, analog gain, and digital gain, controlling lens step focus, and rotating the iris.

Internal processing flow of the ISP firmware (see Figure 2):

Initialization: The ISP control unit, ISP base algorithm library, and 3A algorithm library are initialized, and callback functions are called to obtain sensor initialization parameters.
Dynamic adjustment: The ISP control unit schedules the ISP base algorithm library and 3A algorithm library to perform real-time computation and control. Figure 3 shows the software architecture of the ISP firmware.

Figure 2 Internal processing flow of the ISP firmware

Figure 3 Software architecture of the ISP firmware

Development Mode

The ISP firmware supports various development modes:

Use the self-developed 3A algorithm library.
You need to call the sensor adaptation interfaces provided in the ISP base algorithm unit and 3A algorithm library to adapt to various sensors. You are advised to create a folder for each type of sensor and place the following two files in the folder. You can develop the two files based on the sensor data sheet and seek support from the sensor vendor if necessary.
- sensor_cmos.c
  It is used to implement the callback functions required by the ISP. The callback functions include sensor-specific adaptation algorithms.
- sensor_ctrl.c
  It serves as the underlying driver of sensors and is used to read data from, write data to, and initialize sensors.
Develop your own 3A algorithm library based on 3A algorithm registration interfaces provided by the ISP.
You need to call the sensor adaptation interfaces provided in the ISP base algorithm unit and your own 3A algorithm library to adapt to various sensors.
Use both self-developed 3A algorithm library and your own 3A algorithm library.
For example, use libisp_hiae.a for AE, and your own 3A algorithm library for AWB.

Overall Process

As the front-end capturing unit, the ISP must work with the video input (VI) unit. After the ISP is initialized and configured, the interface timings of the VI must be configured. This ensures that the input timings of different sensors are compatible, and the input timings of the ISP are correct. After timings are configured, the ISP can be started to dynamically adjust the image quality. Then, the output images are captured by the VI and sent for display, encoding, or AI processing.

Figure 4 Flowchart of the ISP firmware

Sample code:

/* Start services such as the VI.*/

......

hi_s32 ret;
hi_isp_3a_alg_lib ae_lib;
hi_isp_3a_alg_lib awb_lib;
hi_isp_pub_attr pub_attr;
pthread_t isp_pid;
hi_vi_pipe vi_pipe = 0;
/* Register the sensor library. */
ret = sensor_register_callback(vi_pipe, &ae_lib, &awb_lib);
if (ret != HI_SUCCESS) {
    printf("register sensor failed !\n");
    return ret;
}

/* Register the AE algorithm library.*/
ae_lib.id = 0;
strncpy(ae_lib.lib_name, HI_AE_LIB_NAME, sizeof(HI_AE_LIB_NAME));
ret = hi_mpi_ae_register(isp_dev, &ae_lib);
if (ret != HI_SUCCESS) {
    printf("hi_mpi_ae_register failed with %#x!\n", ret);
    return ret;
}

/* Register the AWB algorithm library.*/
awb_lib.id = isp_dev;
strncpy(awb_lib.lib_name, HI_AWB_LIB_NAME, sizeof(HI_AWB_LIB_NAME));

ret = hi_mpi_awb_register(isp_dev, &awb_lib);
if (ret != HI_SUCCESS) {
    printf("hi_mpi_awb_register failed with %#x!\n", ret);
    return ret;
}
/* Initialize the external registers of the ISP.*/
ret = hi_mpi_isp_mem_init(vi_pipe);
if (ret != HI_SUCCESS) {
    printf("hi_mpi_isp_mem_init failed with %#x!\n", ret);
    return ret;
}

/* Configure common image attributes.*/
ret = hi_mpi_isp_set_pub_attr(vi_pipe, &pub_attr);
if (ret != HI_SUCCESS) {
    printf("hi_mpi_isp_set_pub_attr failed with %#x!\n", ret);
    return ret;
}
/* Initialize the ISP firmware.*/
ret = hi_mpi_isp_init(vi_pipe);
if (ret != HI_SUCCESS) {
    printf("isp init failed !\n");
    return ret;
}

/* Call hi_mpi_isp_run to separately start a thread.*/
if (0 != pthread_create(&isp_pid, 0, ISP_Run, NULL)) {
    printf("create isp running thread failed!\n");
    return HI_FAILURE;
}

/* Start services such as VPSS.*/

......

/* Stop services such as VI.*/
ret = hi_mpi_isp_exit(vi_pipe);
if (HI_SUCCESS != ret) {
    printf("isp exit failed !\n");
    return ret;
}
pthread_join(isp_pid, 0);
return HI_SUCCESS;

The AE library uses the math library of the standard C library. You need to add the –lm compilation condition to the Makefile.

When using the ISP firmware, ensure that the operating environment is stable.