The Data Made Me Do It

Would you trade your personal data for a peek into the future? Andreas Weigend did.

The former chief scientist of Amazon.com, now directing Stanford University’s Social Data Lab, told me a story about awakening at dawn to catch a flight from Shanghai. That’s when an app he’d begun using, Google Now, told him his flight was delayed.

The software scours a person’s Gmail and calendar, as well as databases like maps and flight schedules. It had spotted the glitch in his travel plans and sent the warning that he shouldn’t rush. When Weigend finally boarded, everyone else on the plane had been waiting for hours for a spare part to arrive.

For Weigend, a fast-talking consultant and lecturer on consumer behavior, such episodes demonstrate “the power of a society based on 10 times as much data.” If the last century was marked by the ability to observe the interactions of physical matter—think of technologies like x-ray and radar—this century, he says, is going to be defined by the ability to observe people through the data they share.

So-called anticipatory systems such as Google Now represent one example of what could result. We’re already seeing the transformations that big data is causing in advertising and other situations where millions of people’s activity can be measured at a time. Now data science is looking at how it can help individuals. Timely updates on a United Airways flight may be among the tamer applications. Think instead of statistical models that tell you what job to take, or alert you even before you feel ill that you may have the flu.

Driving this trend is a swelling amount of personal data available to computers. The amount of digital data being created globally is doubling every two years, and the majority of it is generated by consumers, in the form of movie downloads, VOIP calls, e-mails, cell-phone location readings, and so on, according to the consultancy IDC. Yet only about 0.5 percent of that data is ever analyzed.

“There is so much more data out there that you can afford to tailor it to the individual,” says Patrick Wolfe, a statistician who studies social networks at University College, London. “Statistically, strength comes from pooling people together, but then the icing on the cake is when you individualize the findings.”

For the data refineries of Silicon Valley, like Google, Facebook, and LinkedIn, the merger of big data and personal data has been a goal for some time. It creates tools advertisers can use, and it makes products that are particularly “sticky,” too. After all, what’s more interesting than yourself? Facebook suggests who your friends might be. Google Now gets better the more data you give it.

Exposing more personal data seems inevitable. With the huge jump in sales of smartphones packed with accelerometers, cameras, and GPS, “people have become instrumented to collect and transmit personal data,” says Weigend. And that may just be the start. Already a fringe community of technophiles, known as the quantified-self movement, have been equipping their bodies with sensors, pedometers, even implanted glucose monitors. One we will feature in this month’s MIT Technology Review Business Report is Stephen Wolfram, the creator of the search engine Wolfram Alpha. Wolfram has for years engaged in a massive self-tracking project, cataloguing e-mails, keystrokes, even his physical movements. Wolfram is interested in predictive apps, but also in the insights that large data sets can have on personal behavior, something he calls “personal analytics.” Wolfram’s idea is that just as his search engine tries to organize all facts about the world, “what you have to do in personal analytics is try to accumulate the knowledge of a person’s life.”

The holdup, says Wolfram, is that some of the most useful data isn’t being captured, at least not in a way that’s easily accessible. Part of the problem is technical, a lack of integration. But much data is warehoused by private companies like Facebook, Apple, and Fitbit, maker of a popular pedometer. Now, as the value of personal data becomes more apparent, fights are brewing. California legislators this year introduced a “Right to Know” bill that would require companies to reveal to individuals the “personal information” they store—in other words, a digital copy of every location trace and sighting of their IP address.

The bill is a part of a social movement that is demanding privacy and accountability, but also a different economic arrangement between the people who supply the data and those who apply it. People want more of the direct benefits of big data, and this month’s MIT Technology Review Business Report tracks the technology, apps, and business ideas with which industry is responding.