Engineers at Duke University used machine learning to develop a microscope capable of adapting its lighting angles, colors, and patterns while teaching itself the optimal settings needed to complete a diagnostic task. In a proof-of-concept study, the microscope simultaneously developed a lighting pattern and classification system that allowed it to quickly identify red blood cells infected by the malaria parasite. According to the research team, it performed these tasks more accurately than trained physicians and other machine learning approaches. “A standard microscope illuminates a sample with the same amount of light coming from all directions, and that lighting has been optimized for human eyes over hundreds of years,” professor Roarke Horstmeyer said. “But computers can see things humans can’t. So not only have we redesigned the hardware to provide a diverse range of lighting options, we’ve allowed the microscope to optimize the illumination for itself.” Duke engineers have developed a new type of microscope that uses a bowl studded with LED lights of various colors and lighting schemes produced by machine learning. Courtesy of Roarke Horstmeyer/Duke University. Rather than diffusing white light from below to evenly illuminate the slide, the new microscope features a bowl-shaped light source with LEDs embedded throughout its surface. This configuration allows samples to be illuminated from different angles and with different colors. Because the LED array offers full control over the brightness and the color of each LED, the microscope can provide a variety of illumination patterns, which can be optimized for different sensing tasks. The researchers considered two specific sensing tasks — classifying the malaria parasite within thin blood smears and within thick blood smears — which resulted in two notably different optimal illumination patterns. First they fed the microscope hundreds of samples of malaria-infected red blood cells prepared as thin smears with whole cells. Using a convolutional neural network, the microscope learned which features of the sample were most important for diagnosing malaria and how best to highlight those features. The microscope’s algorithm eventually landed on a ring-shaped LED pattern of different colors coming from relatively high angles. While the resulting images were noisier than a regular microscope image, they highlighted the malaria parasite in a bright spot, and were correctly classified about 90% of the time. Trained physicians and other machine learning algorithms typically perform with about 75% accuracy. The new microscope taught itself the best way to light up red blood cells to spot malaria parasites within. Compared to images produced by a traditional microscope (top), the red blood cell images created by the new microscope (bottom) contain more noise, but the malaria parasites are lit up by bright patches due to the lighting conditions. Malaria-free red blood cells are on the right. Courtesy of Roarke Horstmeyer/Duke University. “The patterns it’s picking out are ring-like with different colors that are nonuniform and are not necessarily obvious,” Horstmeyer said. “Even though the images are dimmer and noisier than what a clinician would create, the algorithm is saying it’ll live with the noise, it just really wants to get the parasite highlighted to help it make a diagnosis.” Horstmeyer sent the LED pattern and sorting algorithm to a collaborator’s lab to see if the results were translatable to different microscope setups. The other laboratory showed a similar outcome. “Physicians have to look through a thousand cells to find a single malaria parasite,” Horstmeyer said. “And because they have to zoom in so closely, they can only look at maybe a dozen at a time, and so reading a slide takes about 10 minutes. If they only had to look at a handful of cells that our microscope has already picked out in a matter of seconds, it would greatly speed up the process.” The researchers also showed that the microscope works well with thick blood smear preparations, in which the red blood cells form a highly nonuniform background and may be broken apart. For this preparation, the machine learning algorithm was successful 99% of the time. According to Horstmeyer, this improvement in accuracy was to be expected, because the tested thick smears were more heavily stained than the thin smears and exhibited higher contrast. However, the thick smears also took longer to prepare. One of the goals of the machine learning microscope project is to cut down on diagnosis times in low-resource settings with a limited number of trained physicians. A group of Duke engineering graduate students has formed a startup company, SafineAI, to miniaturize a reconfigurable LED microscope, a concept that has already earned a $120,000 prize at a local pitch competition. Meanwhile, Horstmeyer is working with a different machine learning algorithm to create a version of the microscope that can adjust its LED pattern to any specific slide it is trying to read. “We’re basically trying to impart some brains into the image acquisition process,” Horstmeyer said. “We want the microscope to use all of its degrees of freedom. So instead of just dumbly taking images, it can play around with the focus and illumination to try to get a better idea of what’s on the slide, just like a human would.” The research was published in Biomedical Optics Express, a publication of The Optical Society (OSA) (www.dx.doi.org/10.1364/BOE.10.006351).