Using deep learning techniques, researchers from the Salk Institute have developed a new microscopy approach that could make microscopic techniques used for brain imaging 16 times faster. The researchers trained their deep learning system using data from the Texas Advanced Computing Center (TACC) at The University of Texas at Austin (UT Austin). Uri Manor, director of the Waitt Advanced Biophotonics Core Facility at the Salk Institute for Biological Studies, believed that deep learning could be used to increase the resolution of microscopic images, similar to the way it has been used to increase resolution in satellite and MRI images. With image analysis specialist Linjing Fang, he developed a way to use graphics processing units (GPUs) to speed up processing of microscopic images. Manor worked with professor Kristen Harris at UT Austin to develop a large collection of data for deep learning training. Using the Maverick supercomputer at the TACC, he created low-resolution analogs of high-resolution microscope images and trained a deep learning network on those image pairs. His first attempts to create superresolution versions of low-resolution images were unsuccessful. “When we tried to test the system on real-world, low-resolution data that was much noisier than our low-resolution training data, the network didn’t do so well,” Manor said. Manor and Fang had been computationally decreasing the resolution of their images for training pairs, but their images were still not low-resolution enough. They were using a type of deep learning architecture called generative adversarial networks (GANs). Jeremy Howard, founder of fast.ai, and Fred Monroe, from the Wicklow AI Medical Research Initiative (WAMRI.ai), came to Salk looking for research problems that could benefit from deep learning. Howard and Monroe recommended to Manor and his team that they add some blur and different kinds of noise to make their images really “crappy.” The Salk researchers crappified their images until they looked similar — and in some cases worse — than a low-resolution image acquired under “real-world” conditions. At the suggestion of Howard and Monroe, the Salk team also switched from a GAN to a U-Net architecture, which the team found easier to train and better at removing noise. Manor retrained his artificial intelligence (AI) system using the new image pairs and deep learning architecture and found that his system could create high-resolution images that were very similar to the ones that had been originally created with greater magnification. Moreover, trained experts were able to find brain cell features in decrappified versions of the low-resolution samples that could not be detected in the original images. An image showing the side-by-side versions of electron microscope captures. Courtesy of the Salk Institute. The researchers put their system to the test by applying their method to images created in other labs with different microscopes and preparations. “Usually in deep learning, you have to retrain and fine-tune the model for different data sets,” Manor said. “We were delighted that our system worked so well for a wide range of sample and image sets.” The success meant that samples could be imaged without risking damage, and that they could be obtained at least 16 times as fast as is possible through traditional methods. “To image the entire brain at full resolution could take over a hundred years,” Manor said. “With a 16 times increase in throughput, it perhaps becomes 10 years, which is much more practical.” According to Manor, his group’s training model is fast and easy to use. Anyone who wants to use the tool will soon be able to log into 3DEM.org, a web-based research platform supported by the National Science Foundation for developing and disseminating new technologies for enhanced resolution microscopy. The Salk researchers have made the code for their tool available through GitHub. Manor hopes to develop software that can do image reconstruction on the fly, so researchers can see superresolution images immediately, rather than in post-processing. He believes there is potential for improving the performance of the millions of microscopes already in use at labs around the world through his tool, and for building a new microscope that takes advantage of AI capabilities. With a proof-of-concept design in place, Manor and his team have developed a tool that could further neuroscience. But without fortuitous collaborations with Harris, Howard, Monroe, and TACC, his tool may never have come to fruition. “You need to have experts open to working together with people from wherever in the world they may be to make something happen,” Manor said. “I just feel so very lucky to have been in a position where I could interface with all of these world-class teammates.” The research was published in bioRxiv (www.dx.doi.org/10.1101/740548).