A recent report compiled by the Insurance Institute for Highway Safety reached a troubling conclusion: Pedestrian injuries and deaths have been on the rise in recent years. This has spurred rapid changes in automobile test protocols, along with the development of pedestrian autonomous emergency braking (PAEB) systems. These systems brake for pedestrians in the path of an automobile and do not depend on the perception of the driver or the reaction of a pedestrian in the car’s path. While existing systems that use cameras and radar can operate well during the day when visibility is good, they fail in the presence of backlighting or in chaotic urban scenes, and when vision is obscured by atmospheric conditions, such as when it is dark outside. Difficult-to-spot pedestrians lurking in the shadows can be seen using the Owl Thermal Ranger system. Courtesy of Owl Autonomous Imaging. “Most safety systems today are centered around regular RGB cameras, which require active illumination or natural light to see a scene,” said Wade Appelman, chief business officer at Owl Autonomous Imaging (Owl AI). “However, they fail in the dark. In fact, over 70% of pedestrian fatalities occur in the dark,” he said. Owl has developed a PAEB that works at night and in inclement weather. The system combines thermal cameras and artificial intelligence to perform the steps required to protect pedestrians. The first of these steps is to detect any warm objects on the road ahead. The second is classifying the object to determine whether it’s a person or an animal that the car should avoid. A pedestrian automatic emergency braking (PAEB) system from Owl Autonomous Imaging incorporates a custom thermal camera and processes inputs using neural networks to detect and identify pedestrians in the road. Courtesy of Owl Autonomous Imaging. Finally, the system initiates an action, sending a message to the driver to stop or avoid the person. If the danger is imminent, the car’s automatic braking system stops the car to prevent a collision. “When imaging pedestrians from a vehicle, two requirements contribute significantly to the likelihood of successful detection — sufficient detail to separate pedestrians from the background and a broad enough field of view to assure that all vulnerable pedestrians are seen,” said Chuck Gershman, CEO and cofounder of Owl AI. “Combining these requirements defines the resolution required in the camera viewing the scene,” he said. Custom sensor The detailed imaging requirements led Owl AI to develop an entirely new sensor for this purpose. In addition to providing tailored coverage and resolution, the new sensor delivers a small image size to minimize optics bulk, proprietary on-chip pixel processing, and wafer-scale packaging. Many of these design considerations aim to make the system accessible at a cost range that permits use in every vehicle. The Owl Thermal Ranger system can detect and discern the difference between cyclists and pedestrians in an urban environment at night. Courtesy of Owl Autonomous Imaging. “As experts in LWIR [longwave-IR] thermal, we knew that if we developed a camera that could be cost effectively produced for the automotive industry, we could change the safety paradigm,” said Appelman. The custom-built longwave-IR camera at the heart of the Owl Thermal Ranger operates in the 8000- to 14,000-nm range and can operate without light or in blinding light, such as oncoming headlights or sun on the horizon. “In many ways, thermal cameras act as if everything that is living is walking around with a retroreflective vest,” Appelman said. The longwave-IR thermal camera addresses the shortcomings in cameras on the market today, notably the high cost. There are calibration challenges that prompted the development of a feature that Owl refers to as shutterless calibration, which ensures that the camera is never “blind.” In addition to the longwave-IR thermal camera (in video graphics array (VGA) resolution), the system includes an Nvidia Orin processor and custom perception software that provides classification, segmentation, fusion, and ranging information. But the secret sauce is in the use of neural networks. “Neural networks are the core of our solution,” Appelman said. “We have the ability to classify objects and provide object bounding boxes in 2D with segmentation, as well as run a ranging convolutional neural network (CNN), which enhances the classified data and provides 3D point cloud ranging information.” Running all the CNNs together allows for object identification with 3D volumetric response point clouds. The proof-of-concept system with a VGA camera is available now. The company is currently working closely with both automotive OEMs and their preferred Tier 1 suppliers to develop custom hardware and software.