Once accessible only to the likes of David Hasselhoff and Batman, autonomous driving is now finding its way into a range of consumer vehicles. Here, in following a car named Junior and its various progeny, we trace the recent history of driverless vehicles and the advances in optics and sensor processing that have made these vehicles
possible.
Sunrise, Nov. 3, 2007. The 11 robot cars
line up in the starting chutes and, one by one, pull onto the course: a makeshift
urban landscape built at the former George Air Force Base in Victorville, Calif.
The autonomous vehicles – no drivers and no remote controls – were tasked
with completing a variety of complex maneuvers, including merging, passing, parking
and negotiating intersections, with interacting with both manned and unmanned cars,
and even determining right of way.
The DARPA Urban Challenge pitted 11 driverless vehicle finalists against one another
in a race to complete a range of maneuvers, including passing, parking and negotiating
intersections. Shown are Stanford Racing Team’s and Team VictorTango’s
(Virginia Tech’s) cars together at an intersection (top) and Stanford Racing
Team’s car approaching a traffic vehicle (above). Photos courtesy of DARPA.
The DARPA Grand Challenge is a prize competition, the primary
goal of which is to fund research to develop autonomous vehicle technology that
will help to keep soldiers off the battlefield. The first two challenges were held
in the Mojave Desert, where course obstacles were mostly rocks and bushes. The 2007
Urban Challenge, however, required much more of the cars.
“This has a component of prediction,” said Michael
Montemerlo, a senior research engineer in the Stanford Artificial Intelligence Lab
(SAIL) in California. The Stanford Racing Team won the 2005 Challenge and was one
of the 11 finalists here. “There are other intelligent robot drivers out in
the world. They are all making decisions. Predicting what they are going to do in
the future is a hard problem that is important to driving. Is it my turn at the
intersection? Do I have time to get across the intersection before somebody hits
me?”
Using an array of sensors – including lidar, radar and cameras
– Junior 2 can identify and track vehicles, bicycles and pedestrians, and
otherwise find its way around town without the aid of a driver. Courtesy of Mike
Sokolsky, Stanford Artificial Intelligence Lab.
By midmorning, almost half of the cars had been removed from the
course, unable to complete the tasks for one reason or another. One, a crowd favorite
from the previous challenge, had started driving erratically and nearly plowed into
the one-time commissary building. Six cars crossed the finish line. In first place:
Tartan Racing, from Carnegie Mellon University in Pittsburgh. In second: Stanford,
with its modified 2006 Volkswagen Passat Wagon named Junior.
Junior achieved its second-place showing thanks to an array of
sensors and positioning systems – not to mention the sophisticated software
developed at SAIL for perception, mapping and planning, giving it the machine-learning
ability needed to improve its driving and maintain a cohesive understanding of what’s
going on all around it.
The sensors included two side-facing light-detection and ranging
(lidar) units made by Sick AG of Waldkirch, Germany, and a forward-facing Riegl
LMS-Q120 lidar made by the Horn, Austria-based Riegl Austria. These allowed the
car to determine its position in real time by locating lane markings based on differences
in brightness in the ground, and to estimate its position to within 5 cm. Additional
lidar units mounted in the front and rear, and on top of the car, aided in object
detection and tracking.
Putting Junior in its place
The potential military applications of autonomous driving technology
are important, but the developers of this remarkable little station wagon were simultaneously
working toward another goal: applying the technology to improve the safety and overall
driving experience in consumer vehicles. To this end, Volkswagen worked closely
with Stanford in developing both Junior and Stanley, the modified 2004 Volkswagen
Touareg that won the 2005 Grand Challenge (and that is on display in the Smithsonian
Institution in Washington).
Late last year, Volkswagen and Stanford University’s School
of Engineering announced the Volkswagen Automotive Innovation Laboratory (VAIL),
the next step in the ongoing relationship between the two organizations. The Volkswagen
Group has dedicated $5.75 million to the creation of the lab, including $2 million
for building construction and an additional $750,000 per year for the next five
years to fund research and teaching activities – encompassing, for example,
far-ranging collaborations between Stanford researchers, international visiting
scholars, automotive equipment manufacturers and Silicon Valley experts.
At the formal dedication and opening of VAIL this past April,
presided over by German Chancellor Angela Merkel, Volkswagen and Stanford demonstrated
their latest step forward in vision-assisted autonomous driving: Junior 3, a blue
Passat running the prototype Autonomous Valet Parking system.
In April of this year, German Chancellor Angela Merkel presided over
the formal dedication and opening of the Volkswagen Automotive Innovation Laboratory
(VAIL), the next step in the ongoing relationship between the car company and Stanford
University’s School of Engineering. Courtesy of Volkswagen Group of America.
Junior 3 can navigate a parking garage and park itself using mostly
stock parts. Both the camera positioned in front of the rear-view mirror and the
front radar system are available as package options from Volkswagen. The off-the-shelf
lidar units attached to the side of the car cannot be purchased as package options,
but the company does offer other, similar side-looking sensors for lane-assist and
blind-spot detection. Drivers can recall the car with the push of a button, using
an iPhone or other smart phone app.
Bloggers and assorted other observers have pointed out the shortcomings
of the system as demonstrated in April: It can park itself only if provided a map
of the garage, and it cannot detect pedestrians who might happen to step in front
of it. Volkswagen envisions garages dedicated to cars with the systems, thus overcoming
these obstacles. Whether or not this comes to pass, Junior 3 demonstrates that,
technologically, vision-assisted driverless parking is within reach.
Big Brother takes to the streets
Also in attendance at the formal dedication of VAIL: Junior
2, the most recent version of the car used in the DARPA Urban Challenge. Although
Junior 3 is a demonstration of the Autonomous Valet Parking system, this car serves
primarily as a test bed for artificial intelligence research in the driving domain,
with the specific goal of developing a fully autonomous car capable of driving on
urban streets in traffic.
“While we’re not going to see fully hands-off cars
on the road anytime soon,” said Mike Sokolsky, a robotics research engineer
at SAIL, “almost all of the research we do could be used to increase safety
and efficiency in driving, even when a human is primarily in control.” Indeed,
a number of options found in consumer cars today were made possible by autonomous
driving research: adaptive cruise control and lane position monitoring, for instance.
Junior 2 identifies and tracks vehicles, bicycles and pedestrians
all around it, taking advantage of the global view afforded by the many sensors
mounted on the car; it never loses focus or forgets to check a blind spot before
changing lanes. But the car still isn’t as adept as human drivers at interpreting
all of the information it receives about its environment. “At this point,
the sensors available to autonomous cars can provide very rich information,”
Sokolsky said, “but making the most of it is definitely still a challenge.”
To date, the car has relied largely on lidar – using it
to produce a detailed three-dimensional point cloud, its primary source of data.
Cameras offer faster frame rates, higher resolution and much lower costs, albeit
without any real depth information. Stereo vision shows some promise but has relatively
limited sensitivity and resolution. And it will be some time before time-of-flight
depth cameras are inexpensive and robust enough to use in an automotive environment.
The Stanford researchers are seeking to address this issue by
using “familiar and easy-to-process” data from the lidar units to learn
more about the images coming out of the cameras, to establish correlations between
the two types of sensors. For example, by extracting 3-D models of moving objects
from the point cloud and matching it to what the camera sees in the same location,
they can obtain accurate information about distance and shape.
“The hope is to be able to get as much or more information
directly out of a camera without the need for depth sensing at all,” Sokolsky
said. “By relying primarily on a camera, data systems like this are much easier
to integrate into commercial vehicles, as well as taking advantage of the benefits
in frame rate and resolution.”
Junior 2: The Technology
The second car in the Junior line uses an array of sophisticated sensors to facilitate its
fully autonomous driving. Currently, the sensor suite centers on a Velodyne HDL-64E
S2 LIDAR that provides 1.3 million returns per second from 64 beams, with a range
of ~100 m. It also includes six Bosch production automotive radar units for tracking
vehicles, especially at longer ranges, and two Sick LD planar lidar scanners for
near-field detection on the sides and rear of the car. Four Point Grey cameras are
used for passive vision: a Ladybug3 spherical vision head mounted just above the
Velodyne with six 2-MP cameras to provide a full view around the car; a pair of
2-MP forward-facing cameras for stereo vision; and a 15-Hz 5-MP forward-looking
color camera.
Stanford’s car, Junior, uses an
array of sensors to negotiate traffic and a variety of obstacles. Shown here is
an image from the car's internal view. The white dots are laser returns from the
Velodyne lidar, the red bars are obstacles, and the blue arrows and orange lines
are lanes. The yellow boxes represent cars Junior is tracking, and the yellow arrows
extending from Junior show the planned trajectory into the intersection. Image courtesy
of Mike Sokolsky, Stanford Artificial Intelligence Lab.