Wearable Camera Enables Human Motion Capture

Researchers at Tokyo Institute of Technology and Carnegie Mellon University developed a human motion capture system called MonoEye, which consists of a single ultrawide fisheye camera mounted on the user’s chest. The system’s simplicity lends itself to a wide range of applications, in sports, medicine, and entertainment.

MonoEye is based on a single ultrawide fisheye camera worn on the user's chest, enabling activity capture in everyday life. Courtesy of the Association of Computer Machinery.

The system features a 280° field of view and is able to capture the user’s limbs, face, and surrounding environment. To achieve robust multimodal motion capture, the system was designed with three deep neural networks capable of estimating 3D body pose, head pose, and camera pose in real time.

The neural networks have been trained with an extensive synthetic data set consisting of 680,000 renderings of people with a range of body shapes, clothing, actions, background, and lighting conditions, as well as 16,000 frames of photorealistic images.

Due to the domain gap between synthetic and real-world data sets, MonoEye’s developers plan to continue expanding their data set with more photorealistic images to improve accuracy. The researchers said they envision that the chest-mounted camera could be transformed into a small, wearable accessory such as a tie clip, brooch, or sports gear.

The research was presented at the 33rd Association of Computer Machinery Symposium on User Interface Software and Technology.