Scientists at the European Molecular Biology Laboratory (EMBL) combined AI algorithms with light-field microscopy and light-sheet microscopy. The hybrid, AI-enhanced approach shortened the time needed for image processing — from days to seconds — and ensured that the resulting images were clear and accurate. Light-field microscopy captures large 3D images that allow researchers to track and measure fine movements at high speeds, such as a fish larva’s beating heart. However, the technique produces massive amounts of data that can take days to process. Although the final images are comprehensive, they can lack resolution. Light-sheet microscopy looks at a single 2D plane of a sample at one time so that researchers can image samples at higher resolution. Compared with light-field microscopy, light-sheet microscopy produces images that are faster to process, though the images are not as comprehensive since they only capture information from a single 2D plane at a time. To combine the benefits of each technique in a single microscopy approach, the EMBL researchers used light-field microscopy to image large 3D samples and light-sheet microscopy to train AI algorithms. The trained algorithms created an accurate 3D image of the sample. Specifically, high-resolution, 2D light-sheet images served as training data and validation for a convolutional neural network (CNN), which then reconstructed the light-field microscopy data, providing an accurate 3D picture of the sample. A representation of a neural network provides a backdrop to a fish larva’s beating heart. Courtesy of Tobias Wuesterfeld. The use of light-sheet microscopy to train the CNN ensured that the AI algorithms worked correctly. “If you build algorithms that produce an image, you need to check that these algorithms are constructing the right image,” EMBL researcher Anna Kreshuk said. To demonstrate its approach, the team imaged medaka (Japanese rice fish) heart dynamics and zebra fish neural activity with volumetric imaging rates up to 100 Hz. In experiments, the researchers showed that the CNN could deliver high-quality 3D-image reconstructions at video-rate throughput. They also showed that the reconstructed images could be further refined based on the high-resolution light-sheet images. “Ultimately, we were able to take the best of both worlds in this approach,” researcher Nils Wagner said. “AI enabled us to combine different microscopy techniques, so that we could image as fast as light-field microscopy allows and get close to the image resolution of light-sheet microscopy.” Researcher Robert Prevedel, whose group developed the hybrid microscopy platform, said that the real bottleneck in building better microscopes often isn’t optics technology, but computation. He and Kreshuk believe that their approach could potentially be modified to work with different types of microscopes, which would allow many different types of specimens to be examined faster and more fully. For example, it could be used to find genes that are involved in heart development or to measure the activity of thousands of neurons at the same time. Next, the researchers plan to explore whether their method could be applied to samples of larger species, including mammals. The research was published in Nature Methods (www.doi.org/10.1038/s41592-021-01136-0).