Researchers from Kaunas University of Technology (KTU) proposed a deep-learning-based method for the three-dimensional human shape reconstruction when the original figure is only partly visible. The method is relatively low cost, provides high compression of the images obtained, and is easily integrated with existing virtual reality tools. The method was developed using a real-world data set. A clinical trial is pending. Currently used 3D image reconstruction solutions for VR contain complex configurations of multiple cameras and require high levels of computational power to process the image. This can make full object reconstruction impractical and prohibitively expensive, the researchers said. Computer scientists at KTU are building technology to enable physicians to monitor patients at their homes using virtual reality tools. Courtesy of Bermix Studios. To address these issues, the team of computer scientists led by Rytis Maskeliunas, chief researcher in the KTU Department of Multimedia Engineering, proposed a deep-learning-based method that can reconstruct a full human posture point cloud from a depth view. They applied a three-stage adversarial deep neural network to deal with depth sensor noise and perform the refining of depth sensor data for full 3D human shape reconstruction. The team used recordings of multiple subjects performing physical rehabilitation exercises as the data set for the experiment. Two commercially available depth cameras were used to film the subjects from the front and from the side. “A camera sees only a part of the image: If it is filming the frontal view, the view from the back is invisible; if something is blocking the view, the camera cannot see what’s behind. Therefore, we employ artificial intelligence which reconstructs the invisible parts of the image,” Maskeliunas said. A five-stage training approach was adopted for training artificial intelligence. Results were validated via expert testimony, which observed that the network reconstructed the result with only a few flaws, mainly near the end of the limbs. The proposed solution is the continuation of several applications that Maskeliunas and his team are currently developing for the medical field. According to Maskeliunas, in health care, the 3D image of the person is crucial when there is a need to diagnose various traumas related to spinal injuries and for various other purposes. “For example, a doctor may ask their patient to perform a simple task, such as touching their nose or rotating their shoulder. To fully see how the person bends, twists, and in how their posture is changing, the physician needs to see them as a three-dimensional subject, to be able to look at them from all the sides and angles,” Maskeliunas said. He added that the availability of the tools and the variety of applications with which the proposed solution can easily be integrated can make the developed method a preferred approach for 3D image reconstruction. The research was published in IEEE Sensors Journal (www.doi.org/10.1109/JSEN.2021.3124451).