On a daily basis, we all use a camera on a daily basis; Whether a dedicated or integrated webcam to hold video conferences, a reflex camera to take pictures or the camera integrated in the smartphone, this element has become almost indispensable in our lives. But, have you ever stopped to think about how to capture what we see in real life in a digital image? Much to do with the ISP, Image Signal Processor , and that is what we are going to talk about next.
Many years ago, those who had a webcam on their PC were privileged, but today they are integrated into spaces as small as the edge of the screen of a laptop or a smartphone. This has been possible because the optics have been greatly reduced while the quality of the images captured has been improved, and this is thanks to a small processor called ISP that all of them incorporate.
What is the ISP (Image Signal Processor) of the cameras
The ISP is a small processor built into the camera (in the case of smartphones it can be built into the SoC), and to understand how it works you should know that pixels are sensitive to light between some sets of wavelengths; essentially, they are independent of color. The way to get a color image is to put a filter on top, usually a bayer pattern color filter, and then interpolate the color of the adjacent pixels. Thus, the camera’s CMOS sensor does not detect red, green and blue for each pixel but rather detects a different color for each of them and it is the ISP processor that guesses the color depending on the one next to it.
This is called a demonstration, and it is the main job of the ISP on camera. In addition, the ISP takes care of all the general tasks of a camera such as autofocus, exposure or white balance, for example. Recently the function of correcting lens imperfections such as vignetting or color shading imparted by an imperfect lens system has been added, along with things like HDR recombination, noise reduction, filters, face detection and conversion between color spaces. .
Obviously, all this last part means that in the event that the camera manufacturer wants it, they can assign the ISP all the image configuration functions that we then use in the software, although they also have the option to make everything either automatic (Apple) or that the options are configured by the user to their liking. This is what allows, for example, that you can blur the background in your Skype video conferences.
Why is this processor important?
As smartphones have become the main device for capturing images and video in people’s daily lives (and in recent times also webcams, even if it is those built into laptops) this series of functions that we consider basic As autofocus has become increasingly important, manufacturers have been forced to improve the ISP and increase the number of tasks it is capable of performing.
Right now, the user experience and camera interface are changing rapidly from generation to generation and more and more serious functions are being integrated in the sense that they require more use of resources (such as the example of blurring the background of Skype that we have put before), so like the controller of an SSD, the ISP of a camera will increasingly play a more important role beyond what the optics are capable of or the Megapixels that have the camera.
For video we must also add one more factor to this equation, the encoder. The way it works is very similar, although the ISP will normally work with subsamples of the camera’s CMOS sensor depending on the sensor’s capabilities. The encoder takes these images and compresses them into a format and bit rate of the choice of the OEM or the user (basically this is currently H.264), and while not all encoders are the same, they should work in conjunction with the ISP to be able to record video.
In the same way, video is being recorded at a higher resolution and with effects added in real time, and this is something that creates a load on the camera’s processor that not everyone is capable of. In short, as technology advances, more functions are added to the cameras and their processor is overloaded, so it is important that their development continues to go according to the evolution of the industry, or we will remain stuck where we are.