Car In-Cabin 3D Sensing Is Within Sight

HANK HOGAN, CONTRIBUTING EDITOR

It is expected that in the future, cars will drive themselves. Between now and then, though, cars will partially drive themselves, much like today’s Tesla with Autopilot. Ensuring that drivers are paying attention during assistance is, and will be, a challenge. Indeed, traffic safety worldwide would benefit if cars could make sure drivers were awake and alert.

Additionally, cars sold in Europe will soon need to ensure that children aren’t accidentally left in parked vehicles. And drivers everywhere may appreciate new options that enable cars to recognize them as unique individuals with personal preferences.

Detection of body and object positions inside a car enables personalized comfort and safety features. 3D sensing could improve detection capabilities. Courtesy of Eyeris.

All these applications could benefit from 3D imaging, aka depth sensing. To capture 3D information, some depth sensing systems project invisible NIR dots onto objects and then measure the distance between the reflected points. The data generated can be used to map the interior of a car, for example, and determine in three dimensions the location of occupants and everything else inside the car. One contending technology to enable 3D sensing employs vertical-cavity surface-emitting lasers (VCSELs).

“As of today, the use of VCSELs in automotive is not really significant, but there will be several opportunities in the next few years. The first one will be related to driver-monitoring systems that will become mandatory in all new cars sold in Europe from 2022,” said Pierrick Boulay, a technology and market analyst with Lyon, France-based Yole Développement. Today more than 70% of the estimated $740 million VCSEL market is driven by consumer applications such as smartphones, according to Boulay. The emerging automotive VCSEL applications will piggyback on the continued cost reductions and increased manufacturing expertise arising from the volume generated by consumer products, he said.

A wide temperature operating range capability could be important if VCSELs are to become the dominant source for in-cabin depth sensing.

Capabilities demonstrated in smartphones, such as the depth-sensing Face ID module from Apple, could help with driver facial recognition in automotive settings, Boulay said. By taking advantage of 3D sensing, vehicle recognition systems may make it possible for preferences in seat and mirror position, temperature, and infotainment to be personalized and automatically adjusted as soon as the system recognizes a driver. (The car could do the same for a passenger, sans mirror adjustment and other items that pertain only to driving.)

The same driver-monitoring technology could check for the presence and health of infants in car seats. This would help prevent the deaths of children in hot cars, of which there were 53 in 2018 in the U.S. and 800 in the previous 20 years, according to the U.S. National Safety Council.

Depth sensing in the cabin

Keeping kids from being left in hot cars is a concern worldwide. According to Angela Suen, director of automotive product line management for laser maker Lumentum Operations LLC of San Jose, Calif., European regulators are pushing to deploy child detection technology, and VCSELs could be used to provide 3D sensing for this application, which requires more than simply imaging an object.

The market for automotive applications of VCSELs today is small, but analysts expect it to grow substantially over the next five years (top), due in part to the use of VCSELs for in-cabin 3D sensing (bottom). CAGR: compound annual growth rate. Courtesy of Yole Développement.

“You have to detect the position of the baby,” Suen said. Such 3D information could be used to reveal movement, which can be an important indication of health and general well-being.

Spotting what’s going on in the back seat of a vehicle also requires the correct placement of depth-sensing light sources and sensors, she said. Thus, VCSELs for child- or passenger-monitoring would be built into the rearview mirror, while driver monitoring would be performed using VCSELs in the steering wheel.

3D sensing could help improve in-cabin applications such as driver monitoring, passenger detection, biometrics, and comfort control.

In both cases, the light sources need to be invisible and eye safe to avoid distracting or harming vehicle occupants. The light source also must be compatible with the imaging sensors, which for cost reasons would be silicon based. These constraints mean the source should be in the NIR and relatively low power.

Lumentum uses 940-nm wavelength VCSELs in its solutions for automotive in-cabin applications, Suen said, adding that the distance from the steering wheel to the driver is about the same as that between a user and a smartphone.

In-cabin automotive applications are easier on a car’s electronics than those outside the vehicle, in part because the operating temperature range is not as wide. Applications outside a car require devices to function at temperatures up to 125 °C, whereas in-cabin electronics may only need to work to 85 °C. That specification is close to the upper temperature limit for consumer goods, which is typically 70 °C. A lower maximum operating temperature is easier for VCSELs to handle, and the laser technology is ready for in-cabin uses, Suen said.

Lumentum is developing solutions for the full temperature range. “It’s the higher temperature that we are working on,” she said.

Alternate sources and methods

A wide temperature operating range capability could be important if VCSELs are to become the dominant source for in-cabin depth sensing. A competing technology that includes a cost advantage is LEDs.

Gesture recognition inside cars, enabled by 3D sensing, could allow drivers to control equipment with the wave of a hand. VCSELs operating in the NIR offer one potential light source for 3D sensing systems. Courtsey of Lumentum.

But LEDs also suffer from several disadvantages when compared to VCSELs, according to Suen. They have a lower beam power density and therefore a shorter range. And unlike lasers, they are not coherent, which affects the data that can be extracted from their beam.

A third drawback is that LEDs have a larger color change in response to temperature changes. LED output moves 0.25 nm per degree Celsius as compared to 0.06 to 0.07 nm per degree for VCSELs. This low rate of change in the laser means that VCSEL output moves 12.5 nm in wavelength over the entire 180° Celsius automotive operating range, which makes the job of the sensor in 3D applications easier when compared to the 45-nm change in output by LEDs. VCSELs also offer better beam quality and faster switching.

3D sensing could help provide access to a variety of functions and capabilities involving safety and comfort, with access enabled by advanced automotive electronics. Courtsey of II-VI.

One 3D sensing method uses structured light, which involves projecting a pattern of dots. To avoid distraction and conserve power, these spots of light may be invisible and only appear in bursts. The dots hit an object and a sensor captures the reflection. The sensor then compares the location of the dots after they bounce off the object to the location where they appeared in the original pattern. The differences between the ideal and the actual provide depth information.

A second technique is time of flight (TOF), which calculates depth from the time it takes for a light pulse to travel to an object and back. This approach is used in lidar and can cover distances of hundreds of meters. Structured light, in contrast, is only good for relatively short distances, such as the few meters, at most, found inside a car cabin. Compared to TOF, structured light (a method based on triangulation) offers higher accuracy, which is a must for facial recognition.

Benefits of 3D

In-cabin applications of VCSELs are seeing a lot of activity, but today they appear only in high-end automobiles, said Gerald Dahlmann, director of strategic marketing for consumer electronics at II-VI Inc. of Saxonburg, Pa. The company is already shipping VCSELs in volume for optical touchpads in steering wheels and other uses, although not yet for depth sensing, he said.

He predicted that TOF sensors will come to the market in volume this year, which will push laser performance and drive advancements. According to Dahlmann, automakers and other OEMs want increased accuracy, higher pixel count, longer range, and lower power consumption out of 3D sensing systems.

For both in-cabin and short-range lidar, a 940-nm wavelength, which II-VI products offer, is preferred, he said. It’s well into the NIR and doesn’t emit a visible red glow. In addition, the sunlight spectrum has a local minimum at this wavelength, hence minimizing interference with 3D imaging from ambient sunlight.

The different 3D sensing techniques impose different requirements on VCSELs, Dahlmann said. In TOF, VCSELs are pulsed, and short rise-and-fall times are critical. The duty cycle is typically high, which means the operating temperature of the device can be at a maximum. II-VI has proprietary design and manufacturing methods that allow its devices to handle this environment, he said.

In contrast, structured light places different constraints on the light source. “For triangulation-based approaches, the uniformity of the dot pattern is most critical,” Dahlmann said. “It requires accurate control of beam parameters of the individual emitters.”

Beyond technology issues, other aspects of 3D sensing must also be worked out. For instance, depth sensing that ensures drivers are alert and paying attention to what’s going on around them benefits public safety and will likely save lives. But there are privacy concerns, particularly if a face can be tied to data, such as places visited or music preferences.

Bans on facial recognition technology, such as one concerning city government operations in San Francisco, have recently been put in place, and other prohibitions on the technology could follow. Concerns about in-cabin facial recognition may also have to be addressed, with car companies perhaps tackling these issues in privacy policies, in addition to conforming to regulations from local or federal laws.

According to Modar Alaoui, CEO of Eyeris Technologies Inc., no matter how such issues are addressed or how depth data is determined, 3D sensing could help improve in-cabin applications such as driver monitoring, passenger detection, biometrics, and comfort control. The Palo Alto, Calif.-based company uses artificial intelligence to analyze the in-vehicle scene and supply the information needed for the system to understand it. This information could then be used to determine the action the system should take, if any.

The company’s technology has been licensed by leading automakers and tier-one parts and systems suppliers. Alaoui said Eyeris’ solution achieves best-in-class performance with standard 2D cameras. He added that it’s even possible, using AI and information about the interior geometry of a car, to estimate 3D information from 2D images.

Nonetheless, Alaoui acknowledged that adding depth data to the image would make detecting the presence of passengers and deciphering what they were doing easier, thereby improving safety by enabling the deployment of personalized safety features. A key development in the rollout of 3D sensing systems will be a drop in their price, which will change how carmakers view the balance between the costs and capabilities of the technology, according to Alaoui

“From an automotive OEM perspective, there is always a cost-performance trade-off,” he said. “We currently offer the best vision AI performance that a 2D camera sensor can enable. 3D sensing is a lot easier to integrate, since its output already provides rich information.”