Within the field of pathology, the upright light microscope is perhaps the most ubiquitous piece of equipment used for examination. Found on every pathologist’s desk, the glass and light device plays a central role in disease diagnosis, displaying the cellular structure of a specimen that will inform the generation of a patient’s report and subsequent treatment. Since its early development, the microscope has undergone little change in optical design, while the optical resolution theoretical limit was already achieved many decades ago. But change is happening, as augmented reality (AR) is being incorporated into the way microscopes are used in the laboratory setting. The optical microscope offers enduring advantages to the pathology workflow. Due to the process by which the human eye recognizes elements in its field of view, and the method in which the eye-brain connection deconstructs images, the optical view of a specimen via the microscope eyepiece is generally easier to decipher than the viewing of a digital image on a screen. Another advantage of the traditional microscope in pathology is the efficiency of the workflow, because the glass slide and specimen are the raw data that will ultimately be analyzed by a trained physician. But despite these and other advantages, the microscope has been largely disconnected from the revolution in digital technology that has swept the health field. While the x-ray has given way to the MRI, and minimally invasive robotic surgery has become commonplace, the analog microscope remains an anomaly. Unlike a blood sample and other laboratory tests that deliver quantitative results, the pathology diagnosis performed on the microscope is essentially done in a “black hole.” It represents a silo of subjectivity, with no effective methodology for using digital resources, applying diagnostic oversight, or quantifying workflow efficacy. Today, the modern pathology laboratory is undergoing an increase in testing, a reduction in the number of pathologists, and cutbacks in reimbursement. The microscope on its own, for all its assistance in detecting cancer and infectious disease, is unable to meet the needs of clinicians working within an increasingly burdened, cost-conscious, and digital health care system. As worldwide demand for pathologists increases, the growing workload and diagnostic subjectivity inherent in the field are driving a search to digitize pathology and, in particular, to leverage the use of analytical software and artificial intelligence (AI). Adding AR to the microscope eyepiece enables pathologists to access image-analysis software directly within the existing microscope-based workflow. By combining the potential of digital health with the proven advantages of the microscope, the partnership of AI and digital pathology may very well be in sight. AI and photonics In recent years, a growing number of AI algorithms have been integrated into health care fields, furthering the evolution of precision medicine. Yet it is only recently that AI has started making headway into pathology, with now hundreds of groups around the world developing tools for the automated detection of cancer. As with all technological advancements, implementation and adoption challenges have emerged, particularly related to the method of deployment. The platform on which pathology AI algorithms are currently being trained and deployed is called whole slide imaging (WSI), a scanning technology that stitches together a digital image of the glass slide and enables a pathologist to view the entire slide on a computer screen. Once a slide is scanned, an experienced pathologist will annotate the digital slide, creating data sets that will be used to train algorithms to identify various lesions. Eventually, these AI algorithms will be deployed on the scanned images in pathology labs. According to AI developers and manufacturers, the clinical benefits of pathology AI include its ability to double-check the manual diagnosis, prescreen slides for definite negatives or positives, and introduce objective results into what is otherwise regarded as the most subjective science. The proposed financial benefit for labs is to free up pathologists’ time and speed up workflow by automating mundane tasks. Despite these potential advantages, most labs find WSI cost prohibitive for routine pathology applications. Because the raw data from which a diagnostic decision is reached exists on a glass slide, the scanning and screen-based viewing process involved in WSI disrupts and slows down the workflow. Compared to an instantaneous view of the optical specimen under the microscope, a typical slide will take approximately a minute to scan. Furthermore, the digitally heavy WSI files, which vary from 2 to 20 GB and contain multiple magnifications, are not easily transferred throughout a lab’s network. For these and other reasons, only a small fraction of clinical pathology labs have implemented WSI for routine diagnostic review, hampering AI deployment. The microscope, with its ideal structure for examination and common placement on pathologists’ desks, could potentially fill the gap. It could act as the deployment platform for AI and image-analysis software, while offering workflow benefits to labs. For example, Augmentiqs is a patent-pending electro-optic accessory module that attaches to and integrates with existing infinity-corrected microscopes. The module connects the pathologist’s microscope to his or her desk computer and transforms the microscope into a smart device. Placed in the optical path between the nosepiece and the eyepiece, the module maintains the optical plane of the microscope, enabling the microscope user to view and work the tissue specimen in the traditional manner, with enhanced capabilities. The module provides real-time access to digital applications as visual AR within the eyepiece, as well as multidirectional communication with remote consultants and viewers. Augmenting the optical path As the light from the sample illuminated by the microscope lamp passes the magnification objective through the tube lens, a beamsplitter siphons off a small percentage of the light to an embedded camera, capturing a live view of the sample as seen within the microscope eyepiece (Figure 1). Figure 1. A diagram of the AR microscope’s optical path. The blue light from the microscope lamp passes through a beamsplitter that siphons off a percentage of the light to an image sensor. The red light represents the AR display flowing through the same beamsplitter and directed toward the eyepiece. The system also detects objective magnification, the slide label, and the relative XY position of the slide on the stage. Courtesy of Augmentiqs. While scientific cameras have been installed with a C-mount and trinocular lens on microscopes for many years, the cameras’ optical field of view is typically limited and potentially vignetted — with the center brighter than the periphery of the image — compared to the view within the eyepiece. The positioning and optical design of an AR system enables a camera to capture a field of view that is very close to the view of the standard eyepiece. Connected to the pathologist’s PC via USB cable, and operating independently of the microscope’s optical path, the system is able to push a continuous feed of image data that can be saved in any open format or viewed remotely via real-time transmission (Figure 2). Figure 2. The electro-optical module is integrated within the existing light microscope. The pathologist interacts with the digital overlay via the computer mouse or other input device of her choice. The image of the specimen, as well as the AR overlay, can also appear on the computer screen. Courtesy of Augmentiqs. This new technology enhances the microscope and pathology workflow by projecting an AR overlay of digital information on top of the optical field of view within the microscope eyepiece. A projection system controlled by the PC sends light into the beamsplitter that is directed into the optical path such that it appears on top of the optical view of the specimen. The optical design of the AR microscope system incorporates a set of filters located in proximity to the beamsplitter, enabling the light stream from the AR projection system to reach the eye, while simultaneously preventing it from interfering with the image sensor. The ability to both image and project on the same Fourier plane (Figure 3) is one of the key components for technology adaptation, as it maintains proper ergonomic height, provides for a high-quality optical view of the specimen in the eyepiece, and solves the challenge of maintaining a high-brightness and high-resolution digital overlay that can be comfortably visible within the microscope eyepiece at multiple illumination levels. Figure 3. A screenshot of quantitative image analysis software deployed in real time within the microscope. The pathologist chooses an area and then uses computer-assisted diagnostic toolsets. The AR enables the pathologist to view results in the microscope as well. Courtesy of Augmentiqs. The entire system has a machine-ready and readable Open API component capable of supporting other pathology applications and algorithms. The AR thereby functions as a computer screen projected onto the live image, accommodating annotations, morphometric calculations, cell counting, or any other open-source or proprietary third-party algorithms within the view of the microscope eyepiece (Figure 4). Figure 4. An image taken via a cellphone held up to the microscope eyepiece to demonstrate the appearance of AR on top of the optical plane of the specimen. Courtesy of Augmentiqs. An additional camera for monitoring the work area adds further functionality, automatically detecting the slide label, the objective magnification, and the precise XY position of the slide and analyzed region of interest. Looking to the future when pathology AI algorithms receive regulatory approval, the system’s ability to detect slide labels could enable pathologists to integrate a patient’s unique medical file into algorithmic parameters, furthering the advancement of pathology toward the vision of personalized medicine. Moreover, the ability to record the exact position of findings and their annotations enables interoperability with whole-slide imaging data collected on other instruments, as well as the ability to easily return to the same position for further analysis or peer review. Changing view of microscope Similar in concept to a smartphone and app store, this new technology functions as a communication and software-deployment platform within the microscope. It enables pathologists to choose from a multitude of software applications for their specific needs. For example, one of the major challenges being faced by labs throughout the world is immediate access to high-level and experienced pathologist consultation. Via this technology, consultant pathologists located in remote locations can use real-time telepathology to view a live image of the specimen on their computer screens. In the case of a frozen section, for example, a pathologist who specializes in a certain tissue type can view an image that is of high digital-pathology grade and similar to the dimensions of the tissue as seen within the PC’s microscope eyepiece. The remote pathologist can then collaborate with the microscope user by making annotations and morphometric calibrations on the computer screen — which appear immediately as AR data within the microscope eyepiece of the PC (Figure 4). This scenario is being played out at Thomas Jefferson University Hospital on an almost daily basis, reducing operating room time by as much as 30 minutes per use of the AR system. A second application that can improve workflow efficiency is the use of open-source image-analysis software for quantitative immunohistochemistry stains. Compared to semiquantitative work performed by the pathologist, quantitative image analysis provides objectivity and plays a critical role in deciding optimal treatment methods. Combined with the ability to display these results as an AR overlay, the University of Pittsburgh Medical Center recently published a study demonstrating achieved time savings. By maintaining the positive qualities of the microscope while integrating digital capabilities, the microscope can, and will, continue to play a primary role in the workflow of pathologists. Meet the author Gabe Siegel is co-founder of Augmentiqs. He earned a bachelor’s degree in accounting and marketing from DePaul University in Chicago, and he developed the concept for Augmentiqs.