KAIST researchers have applied deep learning technology to 3D holographic microscopy, demonstrating the microscopy method’s potential for the practical study of cell interactions. The method holds implications for the study of cancer immunotherapy. Previously, 3D holographic microscopy, or holotomography (HT), had been limited to the study of single cells, though never interactions between cells, due to the difficulty of segmentation, or distinguishing the different parts of a cell, and further, distinguishing between the interacting cells. Manual segmentation, where the user distinguishes the parts by hand, is time-consuming and difficult, making it impractical for pairing with 3D images. To overcome this specifically, the researchers developed automatic segmentation techniques. Here, a computer algorithm interpreted the images. Immune cell dynamics imaged in 3D through the DeepIS method developed at KAIST. Courtesy of professors YongKeun Park and Chan Hyuk Kim, KAIST. “But these basic algorithms often make mistakes,” said YonKeun Park, a professor in KAIST’s Department of Physics, “particularly with respect to adjoining segmentation, which of course is exactly what is occurring here in the immune response we’re most interested in.” Park’s team instead applied a more advanced technology to the task: deep learning. The method is enabled by artificial neural networks based on the way the human brain recognizes patterns. With multiple layers of artificial neural networks, deep learning is able to tackle much larger and unlabeled data sets, with the AI developing its own labels for the concepts it encounters. The method developed at KAIST is called DeepIS, with “IS” referring to “immunological synapse,” the specific process the researchers seek to study. IS is the junction between an immune cell known as a T cell and the cell that presents an antigen, which helps the T cells to recognize and respond to undesirables within the body such as pathogens and cancer cells. The IS is the key process in enabling the immune system to respond to specific types of invaders. DeepIS is able to develop and deploy its own concepts by which it distinguishes the different parts of the IS junction process. To test the method, the team applied it to the dynamics of a particular IS junction between chimeric antigen receptor (CAR) T cells and target cancer cells. Those results were then compared to what would normally have to be done, that is, manual segmentation. The researchers found that not only was DeepIS able to define areas within the IS with a high level of accuracy, but it was also able to capture information about the total distribution of proteins that might not be easily measured with traditional techniques. “In addition to allowing us to avoid the drudgery of manual segmentation and the problems of photo-bleaching and phototoxicity, we found that the AI actually did a better job,” Park said, referring to issues with fluorescence techniques used to study IS. The researchers’ next step will be to combine the technique with methods of measuring the levels of physical force applied by the different parts of the IS junction, such as holographic optical tweezers or traction force microscopy. The research was published in eLife (www.doi.org/10.7554/eLife.49023).