What is ISP (Image Signal Processor)?

Overview of ISP

ISP (Image Signal Processor) is to do post-processing on the signal output by the front-end image sensor, such as noise reduction and HDR correction. At the same time, the ISP can also realize functions such as face recognition and automatic scene recognition.

Although the image signal processor can perform post-processing on the quality of the proofs, different chips or different products using the same chip also have obvious differences in shooting performance. This is because ISP is also has programmable. The new algorithm after debugging in different ways will widen the gap with the integrated original algorithm. At present, most mobile phones on the market integrate ISP, and some products use independent ISP chip.

Take a smartphone as an example. The two main types of data it receives are images and sound. The sound wave input and output, as the initial function of the telephone, has reached perfection in technology since its development, and can only improve the stability of sound quality.

Xiao Mi ISP chip

The image has become the core function of the smartphone, and its shooting technology and image quality are constantly upgrading. The data generated by it also occupies the most memory resources of the mobile phone. It has become the most important data source for various applications.

The image of things enters the eye through the reflection of light, and the photoreceptor cells on the retina can detect the intensity, angle, and position of the light, which triggers bioelectric signals, which are finally transmitted to the brain nerve cells for visual image imaging, which allows us seeing and remembering what has been seen.

Like the realization of biological vision, the processing process of image vision is basically realized according to this process, including the lens that collects the light source, the image sensor that converts light signals into electrical signals, the ISP that digitally processes image electrical signals, and the AI chip that can analyze and identify the image.

In the process of taking pictures and videos with our mobile phones every day, these humble devices are performing huge calculations hundreds of millions of times per second, so that we can have colorful shooting effects.

In the process that image technology has been approaching or even challenging the limits of human vision, the ISP chip plays a vital role. In this article, we are mainly concerned with the technical logic of ISP, the status quo, and the development trend of the ISP technology industry.

Light beauty artist: ISP's operating logic

ISP, the abbreviation of "Image Signal Processor", is a unit used to process the output signal of the front-end image sensor. In simple terms, what the ISP needs to do is to increase the visual acuity level of the "digital eye" to the level of the "human eye", so that the effect when the human eye sees the digital image is as close as possible to the effect when the human eye sees the real scene.

An ISP is actually an SoC core, which contains units such as CPU, SUP IP, IF, etc., which can run various algorithm programs to process image signals in real-time. The control structure of ISP consists of two parts: ISP logic and Firmware running on it. In addition to completing part of the image signal processing, the logic unit can also calculate the real-time information of the current image. Firmware recalculates by obtaining the image statistics of the ISP logic, and feedback controls the Lens, Sensor, and ISP logic to achieve the purpose of automatically adjusting the image quality.

ISP image signal processing

According to the introduction of professionals, the specific processing flow of the ISP is as follows: After the Lens projects the light signal to the photosensitive area of the sensor, the sensor undergoes photoelectric conversion and sends the original image in Bayer format to the ISP. Then the ISP processes the algorithm to output the image of the RGB space domain to the back-end video capture unit. In this process, the ISP controls the ISP logic through the firmware running on it, thereby controlling the Lens and sensor accordingly, and then completing functions such as automatic iris, automatic exposure, and automatic white balance. Among them, the operation of the Firmware is driven by the interruption of the video capture unit. The PQ Tools tool completes the online image quality adjustment of the ISP through the network port or the serial port.

ISP algorithm processing can perform black level compensation, lens shading correction, bad pixel correction, color interpolation, Bayer noise removal, and white balance (AWB) correction for Bayer images, color correction, gamma correction, color space conversion (RGB to YUV), color noise removal and edge enhancement, color and contrast enhancement in YUV color space. Then output the data in YUV or RGB format. And it’s transmitted to the CPU for processing through the I/O interface for image storage and display.

The sharpening, noise reduction and color optimization of the picture that we often said are all processed in the ISP. And the ISP is also responsible for implementing hybrid focusing operations such as phase, laser, and contrast, and providing support for multiple cameras, etc. The important task is an indispensable key part of mobile phone photography.

Mobile phone camera processing

In the end, we saw that the ISP is like a light beauty artist, enabling us to obtain so many image quality adjustments and improvements at the moment the shutter is pressed. We can also actively select various shooting scene modes before shooting. The fine-tuning of various shooting parameters allows everyone to become a master of imaging.

Become the core of mobile terminal image processing: ISP's industrial transition

The emergence of ISP is closely related to the high-quality requirements of humans for captured images.

During NASA's exploration of the moon, it was discovered that the photos taken by the analog camera on the moon were very faint and unclear and could not reflect the real situation. Since then, the CCD image sensor developed in the United States has solved this problem. In order to ensure that the images captured by the CCD can be displayed clearly and truthfully, it is necessary to process the images collected by the CCD, so that the ISP image signal processor was born.

With the widespread application of digital cameras and camcorders, ISP technology has also developed rapidly. In the process of advancing high-definition video and image products, the innovation of ISP technology has played a huge role, and ISP has gradually evolved from the auxiliary technology of high-definition cameras to the key core technology. In order to further improve the image quality, some high-definition surveillance hardware manufacturers have begun to replace the CCD sensor with the CMOS image sensor, so that the ISP processor is built into the CMOS, which saves space and reduces power consumption.

Mobile photography system

Therefore, this innovation enables the high-definition camera market to achieve an upgrade from analog to digital signals. Generally speaking, different ISPs match the CMOS image sensors of different manufacturers. Most video chips with built-in ISP function support CMOS chips well, and can directly connect to the main products of major manufacturers, making the camera smaller in size and product form.

Now, every mobile device equipped with a camera is equipped with an ISP. However, many domestic smartphone manufacturers, usually advertise that their products use Sony's CMOS image sensor, and even give the specific model of this photosensitive element, and also give an exploded view of the multi-lens array. But after consumers get it, the photos taken with it are not as good as expected. One of the big reasons is the ISP. Although these manufacturers have obtained high-quality optical and photosensitive components, the ISP has not been able to keep up.

Technical Barriers of ISP

Now, major smart mobile device chip manufacturers, IP providers, and smartphone manufacturers are launching their own ISP chips, but there are still large technical barriers to ISPs.

First of all, the algorithms of a large number of modules in ISP influence each other, and many of its algorithms require a lot of adjustment work, which requires a lot of long-term experience accumulation.

Secondly, as a commercial ISP, many new functional modules are required, such as X-talk crosstalk, VSM, and other modules, which also increase the technical barriers of ISP.

This makes it easier for leading ISP technology to be mastered by large chip manufacturers and established manufacturers with first-mover advantages and years of accumulation, such as Apple, Qualcomm, Samsung, and MediaTek.

However, more and more mobile phone chip manufacturers have begun to develop their own ISPs and algorithms. Huawei HiSilicon has integrated the self-developed 950 chip from Kirin 950, making Huawei P9 the first camp of the main camera phone.

As a global mobile CPU chip core IP provider, ARM also launched its first ISP chip in 2018, initially for mobile image processing for automotive equipment, and then launched two new ISP chips, Mali-C52 and Mali-C32, for mobile terminals and IoT devices. ARM's ISP has more than 25 steps of processing when processing each pixel, including three key steps of high dynamic range (HDR), noise reduction, and color management, which have a decisive impact on the image output quality.

We have noticed that in the combing of these ISP products, image signal processing, and AI vision processing are developing in concert. The need for both "seeable" and "understandable" image processing has become the key to the development of future visual equipment.

New Opportunity: Symbiosis of Vision Processor and ISP

Artificial intelligence technology has entered the imaging industry as a new element of technological change, making image processing and calculation usher in more challenges. Now in addition to the ISP, it is also necessary to add an AI-capable vision processor (VP) chip.

At present, ISP chips are in a low-speed and stable growth stage, while VP chips are showing explosive growth.

According to market statistics from Yole, the compound annual growth rate (CAGR) of VP chips from 2018 to 2024 is as high as 18%, and it is expected to reach $14.5 billion in 2024. At the same time, ISP chips maintain a low-speed and stable growth trend. The compound annual growth rate from 2018 to 2024 is only 3%, and it is expected to reach 4.2 billion US dollars in 2024.

AI camera processing

In the mobile smart devices and IoT devices we carry, for image processing and calculation, it is not only necessary to realize the “viewing” function, but also to realize the “analysis” function. Therefore, more and more image hardware manufacturers require adding software integration that supports artificial intelligence algorithms, especially deep learning. Therefore, the hardware system allows the imaging hardware system to exceed the limitation of "shooting images" and achieve the ability to "analyze images".

For consumers, the most important feeling is the facial recognition of smartphones and the AI beauty and photo recognition functions in some mobile phone cameras. In many intelligent vision fields, the integration of VP+ISP has brought about a brand-new intelligent revolution.

Therefore, the explosive development of vision processors is parallel to the development of the ISP industry, because if you want to realize the intelligent vision, you must consider choosing a suitable front-end image processing solution. If the output of the front-end image processing is very bad, the result of the visual processing system will also be very bad.

The intelligent vision system must have reliable input to produce reliable results, which is the fundamental reason why it cannot leave a powerful ISP.