Search
Menu
PI Physik Instrumente - Microscope Stages LB ROS 11/24

Deep Learning Enables High-Resolution 3D Images with Low-Speed Cameras

Facebook X LinkedIn Email
A team at Nanjing University of Science and Technology drew on deep learning techniques to achieve high-resolution, high-speed 3D imaging using conventional, low-speed cameras. The new 3D imaging method shows that slow-scan cameras, which are low in cost and high in spatial resolution, can be used for high-speed 3D imaging tasks and can provide both high spatial and high temporal resolution.

The new 3D imaging approach will allow researchers to capture rapid events as they unfold — a necessity for applications in industrial inspection, biomedical research, and many other fields.

Among 3D imaging modalities, fringe projection profilometry (FPP) is a widely used technique, due to its capacity for non-contact, precise, full-field measurements. To improve the imaging speed of FPP, the researchers developed an approach they call deep learning-enabled multiplexed FPP (DLMFPP).

The researchers combined computational imaging and deep learning to encode temporal information in space. This allowed the team to overcome the limitations in hardware speed found in low-cost cameras, and enhance the camera speed without decreasing the pixel resolution or signal to noise ratio (SNR) of the 3D image.
A new 3D imaging paradigm developed researchers at Nanjing University could provide a way for slow-scan cameras to quantitatively study dynamic processes with both high spatial and high temporal resolution. Courtesy of Nanjing University of Science and Technology/Chao Zuo.
A new 3D imaging paradigm developed researchers at Nanjing University could provide a way for slow-scan cameras to quantitatively study dynamic processes with both high spatial and high temporal resolution. Courtesy of Nanjing University of Science and Technology/Chao Zuo.

DLMFPP uses the high temporal resolution capabilities of digital micromirror devices and frequency-domain multiplexing to encode temporal information in one multiplexed fringe pattern. This approach eliminates the physical limitations of the sensor frame rate on 3D imaging speed and allows high-resolution, high-speed 3D imaging at an almost one order of magnitude higher 3D frame rate, using conventional low-speed cameras.

The DLMFPP method uses a sequence of fringe patterns, with varying tilt angles, in its projection strategy. When the projection speed surpasses the speed of the camera, the camera captures a multiplexed image overlaid with the sequence of fringe patterns. DLMFPP temporally embeds the sequence of fringe patterns with different tilt angles into a single multiplexed image.

DLMFPP decodes the image into its original sequence by using deep neural networks embedded with Fourier transform and ensemble learning. This enables high-fidelity decoupling of the multiplexed image into the original sequence. Each spatial carrier fringe records the 3D information of the object at different time points. By arranging each fringe pattern to record the scene at a different time, DLMFPP can achieve up to 9 times temporal superresolution imaging beyond the camera’s frame rate.

Excelitas PCO GmbH - Industrial Camera 11-24 VS MR

The researchers validated the effectiveness and versatility of DLMFPP through experimental demonstrations on different types of transient scenes, including rotating fan blades and a bullet fired from a toy gun. The experiments showed that DLMFPP can achieve high-speed kilohertz 3D imaging with low-speed cameras operating at around 100 Hz, without compromising image resolution.

The DLMFPP method overcomes the physical limitations of imaging detector hardware, enabling slow-scan cameras to quantitatively study dynamic processes with high spatiotemporal resolution. Additionally, the compressive imaging mode of DLMFPP offers several advantages, including low cost, reduced bandwidth and memory requirements, and low power consumption.

Unlike conventional computational imaging technologies, DLMFPP does not rely on spatial encoders or other complex optical modulation hardware. By using a simple optical path, DLMFPP avoids photon losses and can therefore make greater use of optical information to ensure a high SNR in 3D imaging. In practice, the DLMFPP method can be implemented on almost any off-the-shelf FPP system.

3D imaging and sensing have become important research directions in optical metrology and information, and advancements in optoelectronics have spurred interest in capturing and documenting instantaneous phenomena.

Structured light-based 3D imaging speed is limited to the native detector frame rates. One solution has been to increase the camera’s speed, but enhancing camera speed often comes at a cost, such as loss of pixel resolution and SNR in captured images. High-speed cameras capture images at a high frame rate without reducing resolution, but are very expensive.

DLMFPP allows affordable, low-speed cameras to be used in place of high-speed cameras and achieves high-speed 3D imaging without forfeiting image resolution. It represents a new 3D imaging paradigm, using FPP, that could open new avenues for the further development of high-speed and ultrahigh-speed 3D imaging technologies.

The research was published in PhotoniX (www.doi.org/10.1186/s43074-024-00139-2).

Published: November 2024
Glossary
optoelectronics
Optoelectronics is a branch of electronics that focuses on the study and application of devices and systems that use light and its interactions with different materials. The term "optoelectronics" is a combination of "optics" and "electronics," reflecting the interdisciplinary nature of this field. Optoelectronic devices convert electrical signals into optical signals or vice versa, making them crucial in various technologies. Some key components and applications of optoelectronics include: ...
deep learning
Deep learning is a subset of machine learning that involves the use of artificial neural networks to model and solve complex problems. The term "deep" in deep learning refers to the use of deep neural networks, which are neural networks with multiple layers (deep architectures). These networks, often called deep neural networks or deep neural architectures, have the ability to automatically learn hierarchical representations of data. Key concepts and components of deep learning include: ...
superresolution
Superresolution refers to the enhancement or improvement of the spatial resolution beyond the conventional limits imposed by the diffraction of light. In the context of imaging, it is a set of techniques and algorithms that aim to achieve higher resolution images than what is traditionally possible using standard imaging systems. In conventional optical microscopy, the resolution is limited by the diffraction of light, a phenomenon described by Ernst Abbe's diffraction limit. This limit sets a...
multiplexing
The combination of two or more signals for transmission along a single wire, path or carrier. In most optical communication systems this is referred to as wavelength division multiplexing, in which the combination of different signals for transmission are imbedded in multiple wavelengths over a single optical channel. The optical channel is a fiber optic cable or any other standard optical waveguide.
Research & TechnologyeducationAsia-PacificNanjing University of Science and TechnologyImaging3D imagingOpticsoptoelectronicsSensors & DetectorsTest & Measurementcamerasneural networksdeep neural networksBiophotonicsindustrialfringe projection profilometrydeep learningsuperresolutionmultiplexing

We use cookies to improve user experience and analyze our website traffic as stated in our Privacy Policy. By using this website, you agree to the use of cookies unless you have disabled them.