Deep Learning Enables High-Resolution 3D Images with Low-Speed Cameras

A team at Nanjing University of Science and Technology drew on deep learning techniques to achieve high-resolution, high-speed 3D imaging using conventional, low-speed cameras. The new 3D imaging method shows that slow-scan cameras, which are low in cost and high in spatial resolution, can be used for high-speed 3D imaging tasks and can provide both high spatial and high temporal resolution.

The new 3D imaging approach will allow researchers to capture rapid events as they unfold — a necessity for applications in industrial inspection, biomedical research, and many other fields.

Among 3D imaging modalities, fringe projection profilometry (FPP) is a widely used technique, due to its capacity for non-contact, precise, full-field measurements. To improve the imaging speed of FPP, the researchers developed an approach they call deep learning-enabled multiplexed FPP (DLMFPP).

The researchers combined computational imaging and deep learning to encode temporal information in space. This allowed the team to overcome the limitations in hardware speed found in low-cost cameras, and enhance the camera speed without decreasing the pixel resolution or signal to noise ratio (SNR) of the 3D image.

A new 3D imaging paradigm developed researchers at Nanjing University could provide a way for slow-scan cameras to quantitatively study dynamic processes with both high spatial and high temporal resolution. Courtesy of Nanjing University of Science and Technology/Chao Zuo.

DLMFPP uses the high temporal resolution capabilities of digital micromirror devices and frequency-domain multiplexing to encode temporal information in one multiplexed fringe pattern. This approach eliminates the physical limitations of the sensor frame rate on 3D imaging speed and allows high-resolution, high-speed 3D imaging at an almost one order of magnitude higher 3D frame rate, using conventional low-speed cameras.

The DLMFPP method uses a sequence of fringe patterns, with varying tilt angles, in its projection strategy. When the projection speed surpasses the speed of the camera, the camera captures a multiplexed image overlaid with the sequence of fringe patterns. DLMFPP temporally embeds the sequence of fringe patterns with different tilt angles into a single multiplexed image.

DLMFPP decodes the image into its original sequence by using deep neural networks embedded with Fourier transform and ensemble learning. This enables high-fidelity decoupling of the multiplexed image into the original sequence. Each spatial carrier fringe records the 3D information of the object at different time points. By arranging each fringe pattern to record the scene at a different time, DLMFPP can achieve up to 9 times temporal superresolution imaging beyond the camera’s frame rate.

The researchers validated the effectiveness and versatility of DLMFPP through experimental demonstrations on different types of transient scenes, including rotating fan blades and a bullet fired from a toy gun. The experiments showed that DLMFPP can achieve high-speed kilohertz 3D imaging with low-speed cameras operating at around 100 Hz, without compromising image resolution.

The DLMFPP method overcomes the physical limitations of imaging detector hardware, enabling slow-scan cameras to quantitatively study dynamic processes with high spatiotemporal resolution. Additionally, the compressive imaging mode of DLMFPP offers several advantages, including low cost, reduced bandwidth and memory requirements, and low power consumption.

Unlike conventional computational imaging technologies, DLMFPP does not rely on spatial encoders or other complex optical modulation hardware. By using a simple optical path, DLMFPP avoids photon losses and can therefore make greater use of optical information to ensure a high SNR in 3D imaging. In practice, the DLMFPP method can be implemented on almost any off-the-shelf FPP system.

3D imaging and sensing have become important research directions in optical metrology and information, and advancements in optoelectronics have spurred interest in capturing and documenting instantaneous phenomena.

Structured light-based 3D imaging speed is limited to the native detector frame rates. One solution has been to increase the camera’s speed, but enhancing camera speed often comes at a cost, such as loss of pixel resolution and SNR in captured images. High-speed cameras capture images at a high frame rate without reducing resolution, but are very expensive.

DLMFPP allows affordable, low-speed cameras to be used in place of high-speed cameras and achieves high-speed 3D imaging without forfeiting image resolution. It represents a new 3D imaging paradigm, using FPP, that could open new avenues for the further development of high-speed and ultrahigh-speed 3D imaging technologies.

The research was published in PhotoniX (www.doi.org/10.1186/s43074-024-00139-2).