Driverless Cars Are Further Away Than You Think

A silver BMW 5 Series is weaving through traffic at roughly 120 kilometers per hour (75 mph) on a freeway that cuts northeast through Bavaria between Munich and Ingolstadt. I’m in the driver’s seat, watching cars and trucks pass by, but I haven’t touched the steering wheel, the brake, or the gas pedal for at least 10 minutes. The BMW approaches a truck that is moving slowly. To maintain our speed, the car activates its turn signal and begins steering to the left, toward the passing lane. Just as it does, another car swerves into the passing lane from several cars behind. The BMW quickly switches off its signal and pulls back to the center of the lane, waiting for the speeding car to pass before trying again.

Putting your life in the hands of a robot chauffeur offers an unnerving glimpse into how driving is about to be upended. The automobile, which has followed a path of steady but slow technological evolution for the past 130 years, is on course to change dramatically in the next few years, in ways that could have radical economic, environmental, and social impacts.

The first autonomous systems, which are able to control steering, braking, and accelerating, are already starting to appear in cars; these systems require drivers to keep an eye on the road and hands on the wheel. But the next generation, such as BMW’s self-driving prototype, could be available in less than a decade and free drivers to work, text, or just relax. Ford, GM, Toyota, Nissan, Volvo, and Audi have all shown off cars that can drive themselves, and they have all declared that within a decade they plan to sell some form of advanced automation—cars able to take over driving on highways or to park themselves in a garage. Google, meanwhile, is investing millions in autonomous driving software, and its driverless cars have become a familiar sight on the highways around Silicon Valley over the last several years.

The allure of automation for car companies is huge. In a fiercely competitive market, in which the makers of luxury cars race to indulge customers with the latest technology, it would be commercial suicide not to invest heavily in an automated future. “It’s the most impressive experience we can offer,” Werner Huber, the man in charge of BMW’s autonomous driving project, told me at the company’s headquarters in Munich. He said the company aims to be “one of the first in the world” to introduce highway autonomy.

Thanks to autonomous driving, the road ahead seems likely to have fewer traffic accidents and less congestion and pollution. Data published last year by the Insurance Institute for Highway Safety, a U.S. nonprofit funded by the auto industry, suggests that partly autonomous features are already helping to reduce crashes. Its figures, collected from U.S. auto insurers, show that cars with forward collision warning systems, which either warn the driver about an impending crash or apply the brakes automatically, are involved in far fewer crashes than cars without them.

More comprehensive autonomy could reduce traffic accidents further still. The National Highway Traffic Safety Administration estimates that more than 90 percent of road crashes involve human error, a figure that has led some experts to predict that autonomous driving will reduce the number of accidents on the road by a similar percentage. Assuming the technology becomes ubiquitous and does have such an effect, the benefits to society will be huge. Almost 33,000 people die on the roads in the United States each year, at a cost of $300 billion, according to the American Automobile Association. The World Health Organization estimates that worldwide over 1.2 million people die on roads every year.

Meanwhile, demonstrations conducted at the University of California, Riverside, in 1997 and experiments involving modified road vehicles conducted by Volvo and others in 2011 suggest that having vehicles travel in high-speed automated “platoons,” thereby reducing aerodynamic drag, could lower fuel consumption by 20 percent. And an engineering study published last year concluded that automation could theoretically allow nearly four times as many cars to travel on a given stretch of highway. That could save some of the 5.5 billion hours and 2.9 billion gallons of fuel that the Texas Transportation Institute says are wasted by traffic congestion each year.

But such projections tend to overlook just how challenging it will be to make a driverless car. If autonomous driving is to change transportation dramatically, it needs to be both widespread and flawless. Turning such a complex technology into a commercial product is unlikely to be simple. It could take decades for the technology to come down in cost, and it might take even longer for it to work safely enough that we trust fully automated vehicles to drive us around.

German engineering
Much of the hype about autonomous driving has, unsurprisingly, focused on Google’s self-driving project. The cars are impressive, and the company has no doubt insinuated the possibility of driverless vehicles into the imaginations of many. But for all its expertise in developing search technology and software, Google has zero experience building cars. To understand how autonomous driving is more likely to emerge, it is more instructive to see what some of the world’s most advanced automakers are working on. And few places in the world can rival the automotive expertise of Germany, where BMW, Audi, Mercedes-­Benz, and Volkswagen are all busy trying to change autonomous driving from a research effort into a viable option on their newest models.

Shortly after arriving in Munich, I found myself at a test track north of the city getting safety instruction from Michael Aeberhard, a BMW research engineer. As I drove a prototype BMW 5 Series along an empty stretch of track, Aeberhard told me to take my hands off the wheel and then issued commands that made the car go berserk and steer wildly off course. Each time, I had to grab the wheel as quickly as I could to override the behavior. The system is designed to defer to a human driver, giving up control whenever he or she moves the wheel or presses a pedal. And if all else fails, there is a big red button on the dashboard that cuts power to all the car’s computers. I practiced hitting it a few times, and discovered how hard it was to control the car without even the power-assisted steering. The idea of the exercise was to prepare me for potential glitches during the actual test drive. “It’s still a prototype,” Aeberhard reminded me several times.

After I signed a disclaimer, we drove to the autobahn outside Munich. A screen fixed to the passenger side of the dashboard showed the world as the car perceives it: three lanes, on which a tiny animated version of the car is surrounded by a bunch of floating blue blocks, each corresponding to a nearby vehicle or to an obstacle like one of the barriers on either side of the road. Aeberhard told me to activate the system in heavy traffic as we rode at about 100 kilometers per hour. When I first flicked the switch, I was dubious about even removing my hands from the wheel, but after watching the car perform numerous passing maneuvers, I found myself relaxing—to my astonishment—until I had to actually remind myself to pay attention to the road.

The car looked normal from the outside. There’s no place on a sleek luxury sedan for the huge rotating laser scanners seen on the prototypes being tested by Google. So BMW and other carmakers have had to find ways to pack smaller, more limited sensors into the body of a car without compromising weight or styling.

Concealed inside the BMW’s front and rear bumpers, two laser scanners and three radar sensors sweep the road before and behind for anything within about 200 meters. Embedded at the top of the windshield and rear window are cameras that track the road markings and detect road signs. Near each side mirror are wide-angle laser scanners, each with almost 180 degrees of vision, that watch the road left and right. Four ultrasonic sensors above the wheels monitor the area close to the car. Finally, a differential Global Positioning System receiver, which combines signals from ground-based stations with those from satellites, knows where the car is, to within a few centimeters of the closest lane marking.

Several computers inside the car’s trunk perform split-second measurements and calculations, processing data pouring in from the sensors. Software assigns a value to each lane of the road based on the car’s speed and the behavior of nearby vehicles. Using a probabilistic technique that helps cancel out inaccuracies in sensor readings, this software decides whether to switch to another lane, to attempt to pass the car ahead, or to get out of the way of a vehicle approaching from behind. Commands are relayed to a separate computer that controls acceleration, braking, and steering. Yet another computer system monitors the behavior of everything involved with autonomous driving for signs of malfunction.

Impressive though BMW’s autonomous highway driving is, it is still years away from market. To see the most advanced autonomy now available, a day later I took the train from Munich to Stuttgart to visit another German automotive giant, Daimler, which owns Mercedes-Benz. At the company’s research and development facility southeast of the city, where experimental new models cruise around covered in black material to hide new designs and features from photographers, I got to ride in probably the most autonomous road car on the market today: the 2014 ­Mercedes S-Class.

A jovial safety engineer drove me around a test track, showing how the car can lock onto a vehicle in front and follow it along the road at a safe distance. To follow at a constant distance, the car’s computers take over not only braking and accelerating, as with conventional adaptive cruise control, but steering too.

Using a stereo camera, radar, and an infrared camera, the S-Class can also spot objects on the road ahead and take control of the brakes to prevent an accident. The engineer eagerly demonstrated this by accelerating toward a dummy placed in the center of the track. At about 80 kilometers per hour, he took his hands off the wheel and removed his foot from the accelerator. Just when impact seemed all but inevitable, the car performed a near-perfect emergency stop, wrenching us forward in our seats but bringing itself to rest about a foot in front of the dummy, which bore an appropriately terrified expression.

Uncertain road
With such technology already on the road and prototypes like BMW’s in the works, it’s tempting to imagine that total automation can’t be far away. In reality, making the leap from the kind of autonomy in the Mercedes-Benz S-Class to the kind in BMW’s prototype will take time, and the dream of total automation could prove surprisingly elusive.

For one thing, many of the sensors and computers found in BMW’s car, and in other prototypes, are too expensive to be deployed widely. And achieving even more complete automation will probably mean using more advanced, more expensive sensors and computers. The spinning laser instrument, or LIDAR, seen on the roof of Google’s cars, for instance, provides the best 3-D image of the surrounding world, accurate down to two centimeters, but sells for around $80,000. Such instruments will also need to be miniaturized and redesigned, adding more cost, since few car designers would slap the existing ones on top of a sleek new model.

Cost will be just one factor, though. While several U.S. states have passed laws permitting autonomous cars to be tested on their roads, the National Highway Traffic Safety Administration has yet to devise regulations for testing and certifying the safety and reliability of autonomous features. Two major international treaties, the Vienna Convention on Road Traffic and the Geneva Convention on Road Traffic, may need to be changed for the cars to be used in Europe and the United States, as both documents state that a driver must be in full control of a vehicle at all times.

Most daunting, however, are the remaining computer science and artificial-­intelligence challenges. Automated driving will at first be limited to relatively simple situations, mainly highway driving, because the technology still can’t respond to uncertainties posed by oncoming traffic, rotaries, and pedestrians. And drivers will also almost certainly be expected to assume some sort of supervisory role, requiring them to be ready to retake control as soon as the system gets outside its comfort zone.

The relationship between human and robot driver could be surprisingly fraught. The problem, as I discovered during my BMW test drive, is that it’s all too easy to lose focus, and difficult to get it back. The difficulty of reëngaging distracted drivers is an issue that Bryan Reimer, a research scientist in MIT’s Age Lab, has well documented (see “Proceed with Caution toward the Self-Driving Car,” May/June 2013). Perhaps the “most inhibiting factors” in the development of driverless cars, he suggests, “will be factors related to the human experience.”

In an effort to address this issue, carmakers are thinking about ways to prevent drivers from becoming too distracted, and ways to bring them back to the driving task as smoothly as possible. This may mean monitoring drivers’ attention and alerting them if they’re becoming too disengaged. “The first generations [of autonomous cars] are going to require a driver to intervene at certain points,” Clifford Nass, codirector of Stanford University’s Center for Automotive Research, told me. “It turns out that may be the most dangerous moment for autonomous vehicles. We may have this terrible irony that when the car is driving autonomously it is much safer, but because of the inability of humans to get back in the loop it may ultimately be less safe.”

An important challenge with a system that drives all by itself, but only some of the time, is that it must be able to predict when it may be about to fail, to give the driver enough time to take over. This ability is limited by the range of a car’s sensors and by the inherent difficulty of predicting the outcome of a complex situation. “Maybe the driver is completely distracted,” Werner Huber said. “He takes five, six, seven seconds to come back to the driving task—that means the car has to know [in advance] when its limitation is reached. The challenge is very big.”

Before traveling to Germany, I visited John ­Leonard, an MIT professor who works on robot navigation, to find out more about the limits of vehicle automation. ­Leonard led one of the teams involved in the DARPA Urban Challenge, an event in 2007 that saw autonomous vehicles race across mocked-up city streets, complete with stop-sign intersections and moving traffic. The challenge inspired new research and new interest in autonomous driving, but ­Leonard is restrained in his enthusiasm for the commercial trajectory that autonomous driving has taken since then. “Some of these fundamental questions, about representing the world and being able to predict what might happen—we might still be decades behind humans with our machine technology,” he told me. “There are major, unsolved, difficult issues here. We have to be careful that we don’t overhype how well it works.”

Leonard suggested that much of the technology that has helped autonomous cars deal with complex urban environments in research projects—some of which is used in Google’s cars today—may never be cheap or compact enough to be employed in commercially available vehicles. This includes not just the LIDAR but also an inertial navigation system, which provides precise positioning information by monitoring the vehicle’s own movement and combining the resulting data with differential GPS and a highly accurate digital map. What’s more, poor weather can significantly degrade the reliability of sensors, ­Leonard said, and it may not always be feasible to rely heavily on a digital map, as so many prototype systems do. “If the system relies on a very accurate prior map, then it has to be robust to the situation of that map being wrong, and the work of keeping those maps up to date shouldn’t be underestimated,” ­he said.

Near the end of my ride in BMW’s autonomous prototype, I discovered an example of imperfect autonomy in action. We had made a loop of the airport and were heading back toward the city when a Smart car, which had been darting through traffic a little erratically, suddenly swung in front of me from the right. Confused by its sudden and irregular maneuver, our car kept approaching it rapidly, and with less than a second to spare I lost my nerve and hit the brakes, slowing the car down and taking it out of self-driving mode. A moment later I asked Aeberhard if our car would have braked in time. “It would’ve been close,” he admitted.

Despite the flashy demos and the bold plans for commercialization, I sometimes detected among carmakers a desire to hit the brakes and temper expectations. Ralf Herttwich, who leads research and engineering of driver assistance systems at Mercedes, explained that interpreting a situation becomes exponentially more difficult as the road becomes more complex. “Once you leave the highway and once you go onto the average road, environment perception needs to get better. Your interpretation of traffic situations, because there are so many more of them—they need to get better,” he said. “Just looking at a traffic light and deciding if that traffic light is for you is a very, very complex problem.”

MIT’s Leonard, for one, does not believe total autonomy is imminent. “I do not expect there to be taxis in Manhattan with no drivers in my lifetime,” he said, before quickly adding, “And I don’t want to see taxi drivers out of business. They know where they’re going, and—at least in Europe—they’re courteous and safe, and they get you where you need to be. That’s a very valuable societal role.”

I pondered Leonard’s objections while visiting BMW and Mercedes. I even mentioned some of them to a taxi driver in Munich who was curious about my trip. He seemed far from worried. “We have siebten Sinn—a seventh sense,” he said, referring to the instinctive road awareness a person builds up. As he nipped through the busy traffic with impressive speed, I suspected that this ability to cope deftly with such a complex and messy world could prove useful for a while longer.