Vehicle Occupant-Monitoring Tech Captures Free-Space Gestures in 3D

Engineers at the Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB) have developed occupant monitoring technology for autonomous and assisted driving applications. The system, the scientists said, is the first to use image data to draw conclusions about occupant activity.

The basis of the advanced occupant monitoring system is the real-time detection of body pose in 3D. The system, which analyzes how quickly the driver would be able to take control of the vehicle, generates a 3D skeletal model for each occupant captured by cameras using machine learning. The resulting image shows the position of the eyes and head, neck, shoulders, elbows, wrists, torso, pelvis, and upper and lower legs — as soon as they are visible in the camera image.

“The algorithms are able to indicate whether someone asleep, or looking at the street, and even how distracted the person may be,” explained Michael Voit, group manager at Fraunhofer IOSB.

An occupant monitoring system developed and introduced by Fraunhofer IOSB engineers detects the driver and all occupants equally. It recognizes the 3D body pose of all occupants, and analyzes their movement behavior and classifies the activity of each individual person detected. This makes it possible to detect critical situations, such as a driver falling asleep, and to distinguish between different activities and the associated levels of distraction. Courtesy of Fraunhofer IOSB.

The system distinguishes between up to 35 activities, including drinking, eating, sleeping, reading, and making phone calls. For this purpose, the machine learning algorithms process the 3D skeleton recognition of the occupants in combination with object detection and intelligent analysis of the movement behavior of all detected persons.

To this end, the system supports both traditional video cameras and infrared cameras that can see in the dark, as well as 3D cameras that can measure the distance between objects and the camera.

“We can not only detect the activities of the driver, but those of all passengers, too — both in the front and back of the vehicle,” Voit said. “The technology is ready for pilot production. We are already in contact with companies who want to use our technology.”

The researchers designed the system so that data remains local to the vehicle.

“The camera data are analyzed in real time, not saved, and do not leave the vehicle at any point. Personalized models are also not needed for this, so no personal data is collected,” said Pascal Birnstill, senior scientist at Fraunhofer IOSB.

The Fraunhofer team said that the technology will support vehicle manufacturers in their response to the EU regulation that will mandate driver monitoring in automated cars, no matter their level of automation.