Reports from the Heart of the Machine
Editor: Du Wei
This study at the University of California, Irvine, makes us look forward to more advanced color night vision goggles in the future.
In some military blockbusters, soldiers wearing night vision goggles to search for advances seem to be indispensable scenes. Night vision systems that use infrared light to observe in the dark night typically render the view as a monochrome image.
Image source: flir.com
However, in a recent study, scientists at the University of California, Irvine, used deep learning AI technology to devise a new method in which infrared vision helps to see visible colors in a scene in lightless conditions.
Andrew Browne, an engineer, surgeon and vision scientist at the University of California, Irvine, said, "Many parts of the world are color-coded in the way people make decisions, such as signal lights."
Night vision systems are a special case. Night vision systems that use infrared light to illuminate the night usually render the scene only in green, and cannot display colors that are visible in normal light. Some newer night vision systems use ultra-sensitive cameras to amplify visible light, but these cameras can barely show colors in dark environments where there is no light to zoom in.
Andrew Browne
So, in this study, the researchers reasoned that each dye and pigment that gives visible light to an object not only reflects a set of visible wavelengths, but may also reflect a set of infrared wavelengths. Well, if a night vision system capable of recognizing infrared fingerprints of each dye and pigment can be trained, it will be possible to display images using visible light associated with each dye and pigment.
Renderings
At present, the relevant papers have been published in the journal PLOS ONE.
Address of the paper: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0265185
This research is the first step in predicting a human visible spectrum scene from imperceptible near-infrared illumination. The next work could greatly facilitate a variety of applications, such as night vision systems and the study of biological samples sensitive to visible light.
Overview of the study
Humans can sense light in the visible spectrum of 400-700 nm. Some night vision systems use infrared light that is not perceptible to humans to convert the rendered image onto a digital display and finally render a monochrome image in the visible spectrum.
The researchers want to develop an imaging algorithm driven by an optimized deep learning architecture that can use infrared spectral illumination in a scene to predict visible-spectrum rendering in that scene, just as humans perceive it using visible-spectrum light. When humans are in complete "darkness" and only have infrared light, they are able to digitally render visible spectral scenes.
Image processing target. Images displayed using only infrared lighting are compared to visible spectrum images after processing NIR data using deep learning.
Andrew Browne says, "The monochrome camera is sensitive to any photons reflected in the scene it sees. So we use a tunable light source to shine light onto the scene and a monochrome camera to capture the photons that bounce off the scene at all the different illumination colors."
To do this, the researchers used a monochrome camera sensitive to visible and near-infrared light to acquire image datasets of facial print images under multispectral illumination covering standard visible red (604 nm), green (529 nm) and blue light (447 nm), as well as infrared wavelengths (718, 777 and 807 nm). Next, they optimized the convolutional neural network with a U-Net-like architecture to predict visible-spectrum images only from near-infrared images.
Sample image in the Portrait Gallery.
The researchers then paired three infrared images with color images to train an artificial intelligence neural network to predict colors in the scene. After training and improving performance, the neural network was able to reconstruct color images from three infrared images that looked very close to real objects. The figure below shows the true color of the visible spectrum on the left and the color under the blessing of the deep learning algorithm on the right.
"When we increase the number of infrared channels or infrared colors, it provides more data and we can better predict what actually looks very close to the real image," says Andrew Browne. The method we propose in this study can be used to obtain images of three different infrared colors that are invisible to the human eye."
However, the researchers only tested their algorithms and techniques on printed color photos. They are seeking to apply these algorithms and techniques to video and, ultimately, to real-world objects and human subjects.
Reference Links:
https://spectrum.ieee.org/night-vision-infrared
https://www.popsci.com/technology/ai-infrared-night-vision-in-color/
IJCAI 2022 - Neural MMO Massive AI Team Survival Challenge
On April 14th, the "IJCAI 2022-Neural MMO Massive AI Team Survival Challenge" was officially launched, initiated by Hyperparameter Technology, co-sponsored by MIT, Tsinghua University Shenzhen International Graduate School and AIcrowd, a well-known data science challenge platform.
Themed "Finding the Strongest AI Team in the Future Open World", the tournament achieved higher achievements than other contestants by exploring, searching and fighting in Neural MMO's massive multi-agent environment. The game also sets new rules, evaluates the strategic robustness of agents against new maps and different opponents, and introduces cooperation and role division in AI teams, enriching the content of the game and enhancing the fun.