Researchers at Cornell University have developed an optical neural network (ONN) that can filter relevant information from a scene before the visual image is detected by a camera. The method will enable faster, smaller, and more energy-efficient image sensors. Researchers in the lab of assistant professor of applied and engineering physics Peter McMahon demonstrated ONN pre-processors that achieved compression ratios of up to 800 to 1 — the equivalent of compressing a 1600-pixel input to just 2 pixels — while still enabling high accuracy across several representative computer vision tasks. The researchers tested the ONN image sensor with machine vision benchmarks, used it to classify cell images in flow cytometers, and further demonstrated its ability to measure and identify objects in 3D scenes. In a digital system, as opposed to an ONN, images are first saved and then sent to a digital electronic processor that extracts information. Such electronic processing is power-consuming and requires far more time for the data to be processed and interpreted. Left: Doctoral student Mandar Sohoni and postdoctoral researcher Tianyu Wang adjust their research setup that tests the ability of an optical neural network to measure objects in a 3D scene. Courtesy of Charissa King-O’Brien. The researchers instead used an optical neural network-based setup, in which the light coming into the sensor is first processed via a series of matrix-vector multiplications. The matrix-vector multiplications compress data to the minimum size needed: four pixels, in this case, according to postdoctoral fellow Tianyu Wang. The mechanism, Wang said, works similarly to human vision: Humans notice and remember key features of a scene, but not all the unimportant details. “By discarding irrelevant or redundant information, an ONN can quickly sort out important information, yielding a compressed representation of the original data, which may have a higher signal-to-noise ratio per camera pixel,” he said. The researchers also tested reconstructing the original image using the data generated by ONN encoders that were trained only to classify the image. According to Wang, the reconstructed images retained important features, which suggested that the compressed data contained more information than just the classification itself. The result, he said, suggests that with better training and improved models, the ONN could yield more accurate results. The researchers believe their work could have practical applications in fields such as early cancer detection research, where cancer cells need to be isolated from millions or billions of other cells. Using flow cytometry, cells flow rapidly past a detector in a microfluidic flow channel. An ONN that has been trained to identify the physical characteristics of the cancer cells can rapidly detect and isolate those cells instantly. “To generate a robust sample of cells that would hold up to statistical analysis, you need to process probably 100 million cells,” said Mandar Sohoni, a doctoral student at Cornell. “In this situation, the test is very specific, and an optical neural network can be trained to allow the detector to process those cells very quickly, which will generate a larger, better data set.” Sohoni said ONNs can also be useful in situations where very low-power sensing or computing is needed. For example, image sensing on a satellite in space would require a device that uses very little power. In this scenario, the ability of ONNs to compress spatial information can be combined with the ability of event cameras to compress temporal information, since the latter is only triggered when the input signal changes. The work was supported by funding from NTT Research, the National Science Foundation, the Kavli Institute at Cornell, the David and Lucile Packard Foundation, and the Canadian Institute for Advanced Research Quantum Information Science Program. The research was published in Nature Photonics (www.doi.org/10.1038/s41566-023-01170-8).