The Next Must-Have Smartphone Feature

Smartphones can tell you when to depart for the airport to make your flight, provide voice-guided directions on the way there, and route around traffic jams. But if you wanted to get audio directions to a specific counter at the airport, you’d be out of luck. That requires a phone to understand its position in great detail, all on its own—even when it is deep inside a building, beyond the reach of GPS signals. Even the most sophisticated phones can’t do that.

That has long frustrated Google engineer Johnny Lee. Why should he “give up on using my cell phone,” he says, when trying to find things inside large buildings? When I talked to him, he was traveling in Singapore, where he had been visiting big malls—and getting lost. As soon as he walked through the front door, he said, his phone had become “somewhat useless.”

But thanks to Tango, a location-sensing system that Lee is developing at Google, phones are about to get much more useful—especially when they’re indoors. In the works for nearly four years, Tango uses sensors, computer vision, and image processing to give phones a much better comprehension of space and motion. It is built on three core technologies: area learning, depth sensing, and motion tracking. Together, they allow Tango phones to learn, remember, and map areas around them; detect how far they are from the floor or a wall or an object; and understand where they are while moving in three-dimensional space. Tango can do this with centimeter-level accuracy, all without relying on external signals such as GPS, Wi-Fi, or Bluetooth.

And because Tango makes devices so adept at understanding their position, it’s much more than the equivalent of indoor GPS. It will let phones interact with their surroundings or with virtual objects in novel ways. If you’re shopping for a new sofa, Tango-enabled apps from Lowe’s and Wayfair will show you which styles will fit the space available and let you plop 3-D images into your living room to see how they would look. Need to measure a painting so you can frame it, or get the dimensions of a bureau so you can sell it online? Tango can use tracking and 3-D sensing data to calculate the dimensions of objects, no ruler or measuring tape needed. Bored? Fire up Phantogeist, a Tango game in which phantom-like aliens lunge at you from behind walls and under floors. Or launch Dinosaurs Among Us, an app from the American Museum of Natural History that relies on Tango. It lets you place dinosaurs around your house and see facts derived from the museum’s current dinosaur exhibit.

Compare that with Pokémon Go, which caused a craze on smartphones this summer even though its characters didn’t really interact with their environment in a realistic way. If Pokémon Go were on Tango, players would be able to approach the cartoon creatures and circle around them instead of just looking at them straight on, from a distance. The characters would appear larger when in close proximity and could be viewed from multiple angles. They would seem to be part of the landscape rather than hovering in the air.

Tango’s greater powers will lead to entirely new kinds of augmented-reality applications, says Blair MacIntyre, director of the Augmented Environments Lab at Georgia Tech. “Being able to scan the world on a smartphone opens up all types of possibilities, including apps that do things we researchers have only demonstrated in our labs, using really expensive equipment,” he says.

Lenovo will release the first Tango device for consumers—a $499 Android smartphone called the Phab2 Pro—this winter. It comes at a time when we are figuring out the best ways to merge the virtual world with the real one. Of the many potential paths forward, Tango stands out for its wide range of functions and ease of use. This thing you probably haven’t heard of has a good chance of being the next must-have smartphone feature.

Helping devices see

Mobile AR apps have been available for years, but Tango is more advanced because it understands its surroundings rather than simply overlaying images on them. GuidiGO, a French-American startup that makes mobile-device-based guided tours for cultural and historical sites, has been testing Tango to produce phone and tablet tours for museums. The apps direct visitors to particular objects by displaying a path of virtual blue dots on top of the floor. When visitors arrive at a featured work, they see a virtual button that appears to be hovering in the air. Touching that virtual button brings up images that appear like a special effect in a movie. In an ancient Egyptian gallery, it could be an x-ray view of a mummy’s sarcophagus. GuidiGO also makes apps for regular Android phones and tablets, iPhones, and iPads, but cofounder and CEO David Lerman says the company’s Tango-based tours are the most immersive. “It’s like comparing silent films with cinema today,” he says.

Like GPS, Tango requires specialized hardware and software. And both are best appreciated once experienced. As Lee puts it: “If you asked people in 2000 whether or not they needed a GPS in their pocket, most probably would have said no. But once people got GPS in their phones, and apps that made use of it, they were able to discover much more about their environment than they could before. And now most of us wouldn’t consider buying a phone without it.”

To truly become an essential smartphone feature, however, Tango still needs to facilitate great apps. This is a work in progress. Jeff Meredith, who heads Lenovo’s Android business group, says only about 50 “very good” Tango apps will be available in Google’s online Play store when the Phab2 Pro goes on sale.

The hardware Tango requires will also need to be streamlined before it can become anywhere near as ubiquitous as GPS in phones. To support Tango, manufacturers must outfit phones with special equipment, including a depth sensor that uses infrared signals to measure distance to points in space and a fisheye-lens camera for wide-angle viewing and motion tracking. Those components increase a phone’s girth, heft, and cost. (The Phab2 Pro is larger and heavier than Apple’s biggest smartphone.) The extra gear also draws more battery power and generates additional heat that needs to be dissipated. In contrast, phones get GPS functionality from a tiny chip and antenna.

Then again, GPS gear once was much larger, but it shrank over the years as mass adoption drove technological advances. Lee is hoping Tango will follow the same trajectory. His team recently worked with the mobile-chip maker Qualcomm to get its popular Snapdragon processors to support Tango cameras, sensors, and algorithms. Providing phone makers with processors that combine multiple Tango functions in a single chip will save some space inside phones, reduce power consumption, and make the technology easier to deploy.

Ultimately, Tango will be most useful if it also gets baked into things other than phones, from robots to wearables, and if those devices all share data about areas they have mapped. Tango, for example, could allow robots to navigate around the furniture and rooms in your house. Lee envisions a future in which a person could tell a Tango-enabled robot, “Go to the door,” and it would instantly know where to head.

If Tango improves the situational awareness of many of the devices we rely on, this add-on to today’s smartphones will end up creating what Lee calls “a new class of computing.”