Reviewing the history of smart phone camera development in this decade, we can clearly see that camera-related hardware iterations will put forward new demands on image algorithms. I will share with you the positive cycle between camera hardware and image algorithms from three dimensions: the first direction is the change of the number of cameras, from single to multi-camera; the second is the change of the spatial dimension of information recording, from two-dimensional, three-dimensional to higher-dimensional; the last is the change of the spectral dimension, from the previous black-and-white camera to color camera, and then from infrared camera to multi-spectral camera.

• Single camera upgrade for better imaging

Ordinary rear camera, due to the lack of dynamic range of the camera sensor itself, will lead to partial overexposure of the whole picture, but there are still some places that are dark, especially at scenes like sunset or sunset glows. To compensate for the lack of hardware, we have a very good HDR algorithm for improvement to record the whole scenery perfectly.

While taking pictures of night scenes, affected by the lack of hardware sensitivity, the details of the picture and noise is more difficult to reconcile, in this case we give a super night solution, not only to improve the brightness of the picture, but also to remove the noise, so that the picture of the night scene is bright and clean.

The above two are for the rear camera. Next, we’ll discuss the front ones. The front is mainly derived from the pursuit of beauty. We have done a lot of research work to make the front photo look good, combined with end-user research to do a lot of detailed tuning, so that each photo looks more natural and comfortable. This is our large aperture bokeh algorithm, with a dual camera to simulate the effect of SLR bokeh. We are the first algorithm company to lay out the application of dual camera, and we continue to improve the effect while delivering to end customers. We did a better job at the edge of the hair, and the hair was not mistakenly bokeh.

The dual camera is mostly used for people photography, and the triple camera configuration is mostly used for landscape photography, but the placement of the three cameras is bound to have a spatial mismatch, resulting in significant jumps when switching. In order to achieve a smooth transition between cameras, ArcSoft has made a number of targeted optimizations in conjunction with relevant hardware manufacturers to achieve the same switching effect as possible with SLR zoom lenses.

• Algorithm From 2D to 3D

For the sake of portability of smart phones, most cameras always have a limited field of view, and the frame of a single photo is basically fixed. In order to allow users to shoot a wider angle while recording in a single photo, we have introduced the PMK solution (A Panoramic Mosaic technology). This algorithm has been adapted for telephoto, wide-angle and super wide-angle, and is an algorithm product that can be friendly and compatible with camera hardware.

In addition to the expansion of the two-dimensional picture, we also use computer graphics advantage algorithm to reconstruct the picture portrait’s three-dimensional information, for the simulation of studio lighting effect. The effect based on a picture made to simulate the image effect of the light source on the human face, in addition to do on the picture, but also directly on the video. Plis, we use the ToF depth camera to collect the spatial stereo information, and then use the SLAM algorithm to model the space. Similarly, we can also build a 3D model for the human body into an Avatar cartoon character. This cartoon character can perform some physical interactions in our modeled virtual world, like climbing, jumping, collision, etc.

• The Extension of Invisible light and spectrum

As we all know, digital cameras were invented at the beginning to record only the information of light intensity, so only black and white picture came out. It was not until Kodak's engineer Bayer invented the RGB arrangement, color digital cameras became popular, and now this Bayer color array sort is still the mainstream. The smart phone manufacturers are also not limited to the exploration of visible light, they have a keen interest in invisible light and the expansion of the spectrum. More image information of different spectrums helps people to explore and understand the world of invisible light.

In terms of infrared light, ArcSoft has developed a series of matching algorithms for face recognition based on face image information. This algorithm is now able to perform face recognition very efficiently and can also adapt to the scenes and situations when face’s block, and can perform face recognition even with a mask on. We can also use the information from the multispectral camera to calculate the exact color temperature of the whole picture and do a good job of accurate color reproduction.

• Trends and Perspectives

Changes in hardware have introduced a lot of demand for image effect optimization, and ArcSoft has seized the opportunity to quickly launch the corresponding solutions. There are currently three directions.

For a while full-screen has been a challenge that all smart phone manufacturers hope to overcome, from Notch screen, water-drop screen, to hole-dig screen, we have made great efforts to improve the screen ratio, full-screen has become a trend in IT design, but this trend brings big challenge to the front camera photo and video quality, mainly resolution decline, lack of dynamic range, face detail reduction, screen foggy feeling, glaring, etc.

For the under-screen selfie, we are constantly communicating with our customers to come up with our solutions in line with their designs. In addition, there is another technology direction is the front under-screen face 3D unlock. There are already some manufacturers have proposed their hardware solutions, ArcSoft to establish related projects to start docking with them.

The second is a multispectral camera. There are many ways to achieve multispectral information collection, but there are still relatively few that can be miniaturized. The current direction of hardware to achieve imaging spectroscopy has these general categories, modulation demodulation filtering, micro and nano surface narrow band filtering, etc. The application of multispectral information can improve the original picture quality, and can also be used for classification and identification of substances alone, and can also do some applications in health management.

After getting the information of various spatial and spectral dimensions, the information can be used to build the extended reality, which is the third direction we mentioned. Among them, the SLAM technology in the extended reality is very dependent on the deep fusion of various sensors, so to do a good job in the extended reality, not only can rely on software algorithms to achieve, but also need a lot of hardwares to cooperate, in order to make the effect to the next level.

The three directions mentioned earlier: under-screen, multispectral, and extended reality, just correspond to the three directions of computational photography. Under-screen belongs to computational optics, using the computer to restore the light path and get better picture quality. Multispectral belongs to computer vision, using multispectral information to extend the visual range of the human eye. Extended reality is more about using computer graphics technology to build an interactive bridge between virtual and reality.

Computer photography is undoubtedly built on the basis of optical hardware, especially with each iteration of optical hardware, there is always a need for a variety of imaging technologies and features. It is clear that the organic combination of optical imaging systems and computer photography will be the future trend, and in ArcSoft we hope to work together with the industry and academia to catch up with the trend and surpass it.

Related Sub-expo:

Precision Optics Expo & Camera Expo

Optical Components, Optical Materials, Optical Processing Equipment, Optical Coating Technology & Equipment, Optical Lens, Camera Module, Camera Accessories, Camera Manufacturing Equipment, Optical Measuring Instruments, Machine Vision & Industrial Application, AR & VR, Sapphire Manufacturing and Applications, Camera Applications

Media Contact

Ethan.li

ethan.li@cioe.cn