A national group formed by Paul McManamon of Exciting Technology, in conjunction with SPIE, is working to develop standardized tests to evaluate lidar technologies. Despite their crucial role in both autonomous vehicles and advanced driver assistance systems (ADAS), lidars currently lack a standardized measure to describe their performance. The group has sought to address the issue with a three-year effort to develop tests and performance standards.
In a recent paper in the journal Optical Engineering, the team reported the findings of the first-year tests and a briefing outline of the larger three-year plan.
Jeremy Bos, an associate professor at Michigan Technological University (MTU), led tests during the first year of the project, with assistance from Ph.D. student Zach Jeffries. Other authors of the research paper included Charles Kershner from the National Geospatial-Intelligence Agency, who set up a ground truth Riegl lidar for the test, and Akhil Kurup, also of MTU.
Map and photograph of the 200-m test area where the performance of different AV lidars was assessed. Courtesy of Jeffries et al. DOI: 10.1117/1.OE.62.3.031211.
The tests’ objective was to evaluate the range, accuracy, and precision of eight automotive-grade lidars using a survey-grade lidar as a reference. Bos, Jeffries, and the team set up various targets along a 200-m path in an open field in Kissimmee, Fla. One key aspect of the targets that made the tests stand out from previous studies was that they were near-perfect matte surfaces with a calibrated 10% reflectivity across a wide spectrum. The team also measured the ability of the lidars to detect the target among highly reflective road signs.
Test results were, in general, consistent with the values advertised in the manufacturer’s datasheets. However, despite recording a mean precision of 2.9 cm across all tested devices, the distribution of the measured values was not Gaussian. Put simply, there was a non-negligible probability for these devices to report very imprecise values (error greater than 10 cm). In some cases, the measured range deviated from the real value by as much as 20 cm. Another important result was that the reflective road signs impaired the target detection performance of the lidars.
Reflective road signs were placed near the targets to measure the precision and range of automotive lidars under more challenging conditions. Courtesy of Jeffries et al. DOI: 10.1117/1.OE.62.3.031211.
“The advertised range performance of lidars pertains to very specific conditions, and performance degrades significantly in the presence of a highly reflective adjacent object,” Bos said.
The first round of tests provided important insights into the performance differences between different lidars, suggesting that the metrics reported by their respective manufacturers are not reliable. However, there is more work to be done.
“The first-year tests were the simplest of them. In the second year, we will duplicate these tests for the characterized lidars while introducing confusion resulting from other automotive lidars approaching from the opposite direction. Additionally, we will measure the eye safety of lidars,” Bos said. “Finally, in the third year, we will include weather effects as a culmination of the complexity build-up.”
Research from the first year of the project was published in Optical Engineering (www.doi.org/10.1117/1.OE.62.3.031211).