Superresolution Microscopy Analysis Accelerated by Machine Learning
Smart data acquisition and deep learning allow researchers and developers to capture larger sets of images for evaluation and diagnosis.
Jakub Pospisil, Czech Technical University in Prague and Bielefeld University; Thomas Huser, Bielefeld University; and Judith Heidelin, LaVision BioTec GmbH, a Miltenyi Biotec Company
Superresolution optical microscopy — a technology that enables the acquisition of fluorescent micrographs of samples with a resolution well below the optical diffraction limit of ~250 nm — is rapidly evolving. Several methods have been developed during the past two decades that allow for this extension of conventional optical microscopy, and they have substantially contributed to the overall understanding of systems as complex as the specific arrangement of chromatin in cells during interphase, or resolving the inner structure of polymer networks in microgels.
Lymph vessels of a mouse imaged by light sheet fluorescence microscopy, showing progenitor cells (pink), aorta (green), and vein (blue). Courtesy of Friedemann Kiefer, René Hägerling, and Cathrin Dierkes/EIMI.
With this new ability to visualize such biomedical systems on the nanoscale, the scientific community has developed a significantly improved understanding of the inner workings of cellular protein factories. This ability, however, comes at the expense of an ever-increasing need for storage space. To produce higher-resolution optical micrographs, hundreds to thousands of diffraction-limited images, or images at much higher pixel densities, have to be taken, which requires substantially longer acquisition and processing times. This need has limited the implementation of these methods by sectors such as the pharmaceutical industry, where high-throughput screening methods are still the first choice.
Several recent developments, however — such as parallel image processing, smart (content-aware) data acquisition, deep learning approaches to improve the quality of image data, microscope automation, and multiplane image acquisition at higher volumetric rates — are presenting new opportunities and challenges to manufacturers, developers, and users of superresolution optical microscopes.
A three-color structured illumination micrograph (SIM) of a living bone cancer cell with nucleus (blue), mitochondria (green), and cytoskeleton (magenta). The image was acquired with a SIM system using instant image reconstruction. Courtesy of Andreas Marwirth/Bielefeld University.
Superresolution comes of age
Superresolution optical microscopy has been made accessible to the broader biomedical research community during the past decade through a number of commercial implementations — stimulated emission depletion (STED), single-
molecule localization microscopy (SMLM), and structured illumination microscopy (SIM) — by almost all major microscope manufacturers. Despite this high degree of commercial availability, acquiring high-quality superresolved images with these systems remains cumbersome on multiple levels, starting with the special requirements placed on sample preparation.
Sample preparation typically still requires the fixation of cells obtained either from cell cultures or as primary cells from animals or humans. Fixation is needed because usually neither the staining techniques nor the image acquisition process are yet compatible with live cell imaging. Background autofluorescence, light scattering, the quality of the fluorescent labels employed, and the quality of the labeling process itself are all crucial for the ultimate success of obtaining high-quality superresolved images from biological samples. The process of acquiring high-quality images from these samples also still requires the interaction of well-trained and highly experienced workers who understand the optical system and the biological relevance of the samples, as well as the analysis of the resulting data.
Schematics showing several high-resolution, superresolution optical microscopy methods. Courtesy of Jakub Pospisil/Bielefeld University.
Sample preparation can be standardized to some degree by providing premixed consumables together with precise step-by-step protocols — for example, by providing unique fluorescent probes and buffers that are optimized for SMLM, as some suppliers have recently begun to do. Also, standardized preparation protocols for the rapid and reproducible preparation and staining of samples are well established in pathology laboratories, although the scale of sample preparation is certainly different in biomedical research labs, where a much broader selection of immunofluorescent markers may be used. This is partly due to the desire to image living samples and dynamic processes, which are more difficult to prepare and handle than chemically fixed (dead) samples.
Deep learning makes its mark
The image acquisition process is also witnessing increased automation and standardization of acquisition hardware and protocols. Smart microscopes can autonomously obtain raw data over long periods of time1. They can autonomously set proper image acquisition parameters to minimize photo damage to the sample and to maximize contrast as well as spatial resolution.
While the automation of superresolution imaging can be solved by implementing simple feedback loops, the full process will likely see significant benefit from the surge in machine learning. Machine learning routines can help to significantly reduce the number of images required for finding ideal imaging parameters. They can be used to control the microscope and revisit sample locations while optimizing imaging parameters, and to identify locations of interest based on user input. Such procedures have already been successfully used for image reconstruction and analysis in SMLM, but the full integration of these methods into a single, fully automated system is still a challenge.
Machine learning is currently being used to improve the quality of images by denoising, for the content-aware reconstruction of images, and to speed up image acquisition by minimizing sample exposure and acquiring just the bare minimum of images required for their superresolved reconstruction2.
An autofluorescence image of a seahorse, imaged by light sheet microscopy. Courtesy of Uwe Schröer/LaVision BioTec GmbH, a Miltenyi Biotec Company.
These methods do, however, depend on extensive a priori training, and they have to be treated with caution because they can come with inherent problems. Inexperienced users can easily rely on these methods too much, which could lead to assigning content where there is none, such as the artificial generation of biological structures that are not observed in reality, or the creation of image artifacts. However, if implemented with parallel-computing platforms, the fast/instant reconstruction of superresolved microscopy images and their instant display also provides unprecedented new ways for users to interact with their microscope and to control the acquisition process3.
For example, until recently, the substantial post-processing time of superresolution optical microscopy images provided feedback about the outcome of an experiment only after the acquisition process had commenced. It did not allow researchers to react to processes occurring below the diffraction limit — for example, the process of viruses entering cells, or pores opening and closing in living cells. With this instant feedback, researchers can now decide during an experiment to continue the imaging process at a site of interest, which could not be resolved by conventional microscopy and was only unveiled in that context during post-processing.
Improvements in methodology
Another challenge for microscope manufacturers is that superresolution microscopy is still evolving. New methods that can further increase the spatial resolution, reduce the photon dosage placed on the sample, and accelerate the speed with which images are acquired are continuously being developed. An example of this progress is the use of new optical systems (diffractive optical elements or prisms) that enable the acquisition of images from various vertical sample planes simultaneously.
One major drawback of many superresolution methods is their limitation to two dimensions. So far, very few solutions have been developed that can expand superresolution imaging methods to deep 3D imaging. An attractive workaround for this limitation is the combination of superresolution imaging methods — in particular, SMLM combined with light sheet microscopy, which provides the possibility of visualizing large sample volumes (up to 5 cm3) at cellular resolution.
In light sheet microscopy, often also referred to as selective plane illumination microscopy, the sample is excited from the side by one or more focused light sheets while the emitted fluorescence light is detected by a camera perpendicular to the illumination plane. 3D image stacks are generated sequentially by moving the sample through the light sheet. Illuminating only the focal plane of the detection objective lens enables camera-based 3D microscopy at high frame rates. Because only the actually observed section is illuminated, photodamage and fluorophore bleaching are kept to a minimum.
With the fully automated light sheet microscope Blaze from LaVision BioTec, three-dimensional image data sets with cellular resolution can be acquired via software-based image reconstruction and image restoration, such as denoising and dynamic contrast enhancement.
The first step for 3D superresolution imaging is to prepare the sample for light sheet microscopy by optical clearing and fluorescent labeling of the sample. The 3D data set acquired with the light sheet microscope is used to find single cells or a volume of interest. The second step is then to cut the volume of interest into thin slices and prepare the slices for superresolution imaging. The correlation of the two microscopy data sets adds the 3D bulk information of the sample location to 2D/3D superresolution images.
A problem that researchers and scientists often face is the high photon dose that current superresolution microscopes place on the sample, which leads to rapid photobleaching or otherwise detrimental effects on sample health. Recent developments such as the combination of patterned illumination — for example, in the form of doughnut-like illumination spots or wide-field illumination with sinusoidal intensity distributions — and single-molecule localization can now determine the position of fluorescent molecules to very high precision on the nanometer scale by exploiting precise knowledge of the illumination function.
The UltraMicoscope Blaze is a light sheet microscope for imaging large-volume samples with cellular resolution. Courtesy of LaVision BioTec, a Miltenyi Biotec Company.
Not only does this method provide even higher spatial resolution, it also allows researchers to do so with even lower sample light exposure by using the minima of the patterned illumination function rather than the maxima, which is used in current microscopes. This method is most effective when only the fluorophore is illuminated, the position of which is ideally determined by using the lowest possible excitation power, which significantly prolongs the duration in which a fluorophore can be imaged4. Such implementation, however, requires a priori knowledge — for example, about the precise shape of the illumination spot or pattern. With these improvements, an isotropic resolution of approximately 2 to 3 nm has recently been demonstrated5.
The future continues to look bright for superresolution microscopy, and a great number of biomedical problems are eagerly awaiting these latest developments. Machine learning will likely leave a permanent mark on the field of microscopy. Machine learning techniques such as denoising and sample-specific acquisition parameters will lead to a significant reduction in the photon dosage that biological samples receive. The large number of superresolution microscopy methods will either supplement each other based on the specific pros of each approach, or they will ultimately be consolidated. All of these developments are likely to further spread the availability and the affordability of superresolution microscopy.
Meet the authors
Jakub Pospisil is a doctoral student in electronics at Czech Technical University (CTU) in Prague. He has been working in the field of superresolution microscopy for more than four years. Currently, Pospisil works on the development of superresolution instruments in Thomas Huser’s lab at Bielefeld University in Germany. In 2015, he received his master’s degree from the Communications, Multimedia and Electronics program at CTU.
Thomas Huser is professor of physics at Bielefeld University. He has 20 years of experience in running independent research groups at research institutions and universities. He studied solid-state physics and holds a doctoral degree in physics from the University of Basel in Switzerland.
Judith Heidelin is project manager for grants and collaboration projects and is part of the intellectual properties/patents team at LaVision BioTec GmbH, a Miltenyi Biotec Company. She has 11 years of experience in research and development of optical systems. She studied physics at Leibniz University of Hannover in Germany and holds a doctoral degree in physics from Laser Zentrum Hannover eV, also in Germany.
Acknowledgment
The authors are supported by the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie Grant Agreement No. 766181, project DeLIVER.
References
1. A.E.S. Barentine et al. (Submitted for publication). 3D multicolor nanoscopy at 10,000 cells a day. bioRxiv, www.doi.org/10.1101/606954.
2. M. Weigert et al. (2018). Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat Methods, Vol. 15, pp. 1090-1097.
3. A. Markwirth et al. (2019). Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction. Nat Commun, Vol. 10, p. 4315.
4. F. Balzarotti et al. (2017). Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes. Science, Vol. 355, pp. 606-612.
5. K.C. Gwosch et al. (2020). MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells. Nat Methods, Vol. 17, pp. 217-224.
/Buyers_Guide/LaVision_BioTec_GmbH/c20348
Published: September 2020
Glossary
- superresolution
- Superresolution refers to the enhancement or improvement of the spatial resolution beyond the conventional limits imposed by the diffraction of light. In the context of imaging, it is a set of techniques and algorithms that aim to achieve higher resolution images than what is traditionally possible using standard imaging systems. In conventional optical microscopy, the resolution is limited by the diffraction of light, a phenomenon described by Ernst Abbe's diffraction limit. This limit sets a...
- deep learning
- Deep learning is a subset of machine learning that involves the use of artificial neural networks to model and solve complex problems. The term "deep" in deep learning refers to the use of deep neural networks, which are neural networks with multiple layers (deep architectures). These networks, often called deep neural networks or deep neural architectures, have the ability to automatically learn hierarchical representations of data.
Key concepts and components of deep learning include:
...
- fluorescence
- Fluorescence is a type of luminescence, which is the emission of light by a substance that has absorbed light or other electromagnetic radiation. Specifically, fluorescence involves the absorption of light at one wavelength and the subsequent re-emission of light at a longer wavelength. The emitted light occurs almost instantaneously and ceases when the excitation light source is removed.
Key characteristics of fluorescence include:
Excitation and emission wavelengths: Fluorescent materials...
- structured illumination microscopy
- Structured illumination microscopy (SIM) is an advanced optical imaging technique used in microscopy to enhance the resolution of images beyond the diffraction limit imposed by traditional light microscopy. The diffraction limit is a fundamental limitation that restricts the ability to distinguish fine details in the microscopic structures.
SIM achieves improved resolution through a process of illuminating the specimen with a patterned light, typically a grid or a stripe pattern. This...
- machine learning
- Machine learning (ML) is a subset of artificial intelligence (AI) that focuses on the development of algorithms and statistical models that enable computers to improve their performance on a specific task through experience or training. Instead of being explicitly programmed to perform a task, a machine learning system learns from data and examples.
The primary goal of machine learning is to develop models that can generalize patterns from data and make predictions or decisions without being...
Featuressuperresolutionoptical microscopydeep learningfluorescencesmart data acquisitionstimulated emission depletionsingle-molecule localization microscopystructured illumination microscopymachine learninglight sheet microscopyLaVision BioTec GmbHMicroscopy