Wearable Camera Devices Enable Human Motion Capture, Bare-hand Interactions

Researchers from Tokyo Institute of Technology, Carnegie Mellon University, the University of St. Andrews, and the University of New South Wales have developed a wrist-worn device that is able to estimate 3D hand poses for virtual reality applications.

The system is the first of its kind to estimate 3D hand poses using a camera focusing on the back of the hand. Courtesy of ACM, UIST.

The system consists of a camera that images the back of the hand and that is supported by a neural network called DorsalNet. The network is able to accurately recognize dynamic gestures, helping the device capture hand motions in mobile settings. According to the researchers, the system outperforms previous attempts at similar devices, with an average of 20% higher accuracy in recognizing dynamic hand gestures, and it achieves 75% accuracy in detecting 11 different grasp types.

Preliminary tests of the device showed that it’s possible to use the system for smart device control and for use as a virtual mouse or keyboard, supporting the scientists' belief that the research development could advance development of controllers supporting bare-hand interactions.

The technique has applications in augmented and virtual reality devices, too, which often require the use of data gloves that can hinder movement or controllers with limited sensing capabilities.

Future improvements to the camera system, such as higher frame rates and light sensitivity, will be needed for real-world use. The researchers said they further envision the incorporation of their device into a smart watch.

The research was presented at the 33rd Association of Computer Machinery Symposium on User Interface Software and Technology.