A bio-inspired sensor developed by researchers at Aalto University can recognize moving objects in a single frame from a video and successfully predict where they will move to. The sensor has use in dynamic vision sensing, automatic inspection, industrial process control, robotic guidance, and autonomous driving technology, among other fields.
Current motion detection systems require a variety of components and complex algorithms performing frame-by-frame analyses, making them inefficient and energy-intensive. Inspired by the human visual system, the Aalto researchers developed a neuromorphic vision technology that integrates sensing, memory, and processing in a single device that can detect motion and predict trajectories.
Conventional sensors capture only a single moment in a frame, but the new sensor can read information about the past and use that to predict the future. Courtesy of Hongwei Tan/Aalto University.
At the core of the technology is an array of photomemristors — electrical devices that produce electric current in response to light. The current doesn’t immediately stop when the light is switched off. Instead, it decays gradually, meaning that photomemristors can effectively “remember” whether they’ve been exposed to light recently. As a result, a sensor made from an array of photomemristors doesn’t just record instantaneous information about a scene like a camera does, but also includes a dynamic memory of the preceding instants.
“The unique property of our technology is its ability to integrate a series of optical images in one frame,” said Hongwei Tan, the research fellow who led the study. According to Tan, the information that each image contains is embedded in the images that follow as “hidden” information, so that the final frame in a video also contains information about all the previous frames.
This quality, he said, allows the researchers to detect motion earlier in the video by analyzing only the final frame using a simple artificial neural network.
To demonstrate the technology, the researchers used videos showing the letters of a word one at a time. Because all the words ended with the letter “E,” the final frame of all the videos looked similar. Conventional vision sensors couldn’t tell whether the “E” on the screen had appeared after the other letters in “APPLE” or “GRAPE.” The photomemristor array used the “hidden” information in the final frame to infer which letters had preceded it and predict what the word was with nearly 100% accuracy.
A sensor made of an array of photomemristors. Courtesy of Hongwei Tan/Aalto University.
In another test, the researchers showed the sensor videos of a simulated person moving at three different speeds. The system recognized motion by analyzing a single frame and correctly predicted the next frames.
Accurately detecting motion and predicting where an object will be are vital for self-driving technology and intelligent transport. Autonomous vehicles need accurate predictions of how cars, bikes, pedestrians, and other objects will move in order to guide their decisions. By adding a machine learning system to the photomemristor array, the researchers showed that their integrated system can predict future motion based on in-sensor processing of an all-informative frame.
In addition, the in-frame information that the researchers attain using photomemristors avoids redundant data flows, enabling energy-efficient decision-making in real time, professor Sebastiaan van Dijken said.
The research was published in Nature Communications (www.doi.org/10.1038/s41467-023-37886-y).