Researchers from Rice University and University of Maryland have created full-motion video technology that could enable cameras to peer through fog, smoke, driving rain, murky water, skin, bone, and other media that reflect scattered light and obscure objects from view. According to Ashok Veeraraghavan, professor of electrical and computer engineering at Rice University, imaging through scattering media is the “holy grail problem” in optical imaging. “Scattering is what makes light — which has lower wavelength, and therefore gives much better spatial resolution — unusable in many, many scenarios. If you can undo the effects of scattering, then imaging just goes so much further.” Veeraraghavan’s lab collaborated with the research group of Maryland co-corresponding author Christopher Metzler to create a technology they named NeuWS, which is an acronym for “neural wavefront shaping,” the technology’s core technique. “If you ask people who are working on autonomous driving vehicles about the biggest challenges they face, they’ll say, ‘Bad weather. We can’t do good imaging in bad weather,’” Veeraraghavan said. “They are saying ‘bad weather,’ but what they mean, in technical terms, is light scattering. If you ask biologists about the biggest challenges in microscopy, they’ll say, ‘We can’t image deep tissue in vivo.’ They’re saying, ‘deep tissue’ and ‘in vivo,’ but what they actually mean is that skin and other layers of tissue they want to see through are scattering light. If you ask underwater photographers about their biggest challenge, they’ll say, ‘I can only image things that are close to me.’ What they actually mean is light scatters in water, and therefore doesn’t go deep enough for them to focus on things that are far away. “In all of these circumstances, and others, the real technical problem is scattering,” Veeraraghavan said. In experiments, camera technology called NeuWS, which was invented by collaborators at Rice University and University of Maryland, was able to correct for the interference of light-scattering media between the camera and the object being imaged. The top row shows a reference image of a butterfly stamp (left), the stamp imaged by a regular camera through a piece of onion skin that was approximately 80 µm thick (center), and a NeuWS image that corrected for light scattering by the onion skin (right). The center row shows reference (left), uncorrected (center), and corrected (right) images of a sample of dog esophagus tissue with a 0.5° light diffuser as the scattering medium. The bottom row shows corresponding images of a positive resolution target with a glass slide covered in nail polish as the scattering medium. Close-ups of inset images from each row are shown for comparison. Courtesy of Veeraraghavan Lab/Rice University. NeuWS could potentially be used to overcome scattering in those scenarios and others. Conceptually, NeuWS is based on the principle that lightwaves are complex mathematical quantities with two key properties that can be computed for any given location. The first, magnitude, is the amount of energy the wave carries at the location, and the second, phase, is the wave’s state of oscillation at the location. Veeraraghavan and Metzler said measuring phase is critical for overcoming scattering, but it is impractical to measure directly because of the high frequency of optical light. Instead, they measure incoming light as “wavefronts,” single measurements that contain both phase and intensity information, and use backend processing to quickly decipher phase information from several hundred wavefront measurements per second. “The technical challenge is finding a way to rapidly measure phase information,” said Metzler, an assistant professor of computer science at Maryland and a Rice alum. Metzler was at Rice University during the development of an earlier iteration of wavefront-processing technology called WISH that Veeraraghavan and colleagues published in 2020. “WISH tackled the same problem, but it worked under the assumption that everything was static and nice,” Veeraraghavan said. “In the real world, of course, things change all of the time.” With NeuWS, he said, the idea is to not only undo the effects of scattering, but to undo them fast enough so the scattering media itself doesn’t change during the measurement. “Instead of measuring the state of oscillation itself, you measure its correlation with known wavefronts,” Veeraraghavan said. “You take a known wavefront, you interfere that with the unknown wavefront, and you measure the interference pattern produced by the two. That is the correlation between those two wavefronts.” Rice University Ph.D. student Haiyun Guo and professor Ashok Veeraraghavan in the Rice Computational Imaging Laboratory. Guo, Veeraraghavan, and collaborators at University of Maryland have created full-motion video camera technology that corrects for light scattering and has the potential to allow cameras to film through fog, smoke, driving rain, murky water, skin, bone, and other light-penetrable obstructions. Courtesy of Brandon Martin/Rice University. Metzler used the analogy of looking at the North Star at night through a haze of clouds. “If I know what the North Star is supposed to look like, and I can tell it is blurred in a particular way, then that tells me how everything else will be blurred.” Veerarghavan said, “It’s not a comparison, it’s a correlation, and if you measure at least three such correlations, you can uniquely recover the unknown wavefront.” State-of-the-art spatial light modulators can make several hundred such measurements per minute, and Veeraraghavan, Metzler, and colleagues showed they could use a modulator and their computational method to capture video of moving objects that were obscured from view by intervening scattering media. “This is the first step, the proof-of principle that this technology can correct for light scattering in real time,” said Rice’s Haiyun Guo, one of the study’s lead authors and a Ph.D. student in Veeraraghavan’s research group. In one set of experiments, for example, a microscope slide containing a printed image of an owl or a turtle was spun on a spindle and filmed by an overhead camera. Light-scattering media were placed between the camera and target slide, and the researchers measured NeuWS’ ability to correct for light scattering. Examples of scattering media included onion skin, slides coated with nail polish, slices of chicken breast tissue, and light-diffusing films. For each of these, the experiments showed that NeuWS could correct for light scattering and produce clear video of the spinning figures. “We developed algorithms that allow us to continuously estimate both the scattering and the scene,” Metzler said. “That’s what allows us to do this, and we do it with mathematical machinery called neural representation that allows it to be both efficient and fast.” NeuWS rapidly modulates light from incoming wavefronts to create several slightly altered phase measurements. The altered phases are then fed directly into a 16,000-parameter neural network that quickly computes the necessary correlations to recover the wavefront’s original phase information. “The neural networks allow it to be faster by allowing us to design algorithms that require fewer measurements,” Veeraraghavan said. Metzler said, “That’s actually the biggest selling point. Fewer measurements, basically, means we need much less capture time. It’s what allows us to capture video rather than still frames.” The work was supported by the Air Force Office of Scientific Research, the National Science Foundation, and the National Institutes of Health, and partial funding for open access was provided by the University of Maryland Libraries’ Open Access Publishing Fund. The research was published in Science Advances (www.doi.org/10.1126/sciadv.adg4671).