The Next Big Step for AI? Understanding Video

For a computer, recognizing a cat or a duck in a still image is pretty clever. But a stiffer test for artificial intelligence will be understanding when the cat is riding a Roomba and chasing the duck around a kitchen.

MIT and IBM this week released a vast data set of video clips painstakingly annotated with details of the action being carried out. The Moments in Time Dataset includes three-second snippets of everything from fishing to break-dancing.

“A lot of things in the world change from one second to the next,” says Aude Oliva, a principal research scientist at MIT and one of the people behind the project. “If you want to understand why something is happening, motion gives you lot of information that you cannot capture in a single frame.”

The current boom in artificial intelligence was sparked, in part, by success in teaching computers to recognize the contents of static images by training deep neural networks on large labeled data sets (see “The Revolutionary Technique That Quietly Changed Machine Vision Forever”).

AI systems that interpret video today, including the systems found in some self-driving cars, often rely on identifying objects in static frames rather than interpreting actions. On Monday Google launched a tool capable of recognizing the objects in video as part of its Cloud Platform, a service that already includes AI tools for processing image, audio, and text.

The next challenge may be teaching machines to understand not just what a video contains, but what’s happening in the footage as well. That could have some practical benefits, perhaps leading to powerful new ways of searching, annotating, and mining video footage. It also figures to give robots or self-driving cars a better understanding of how the world around them is unfolding.

The MIT-IBM project is in fact just one of several video data sets designed to spur progress in training machines to understand actions in the physical world. Last year, for example, Google released a set of eight million tagged YouTube videos called YouTube-8M. Facebook is developing an annotated data set of video actions called the Scenes, Actions, and Objects set.

Olga Russakovsky, an assistant professor at Princeton University who specializes in computer vision, says it has proved difficult to develop useful video data sets because they require more storage and computing power than still images do. “I’m excited to play with this new data,” she says. “I think the three-second length is great—it provides temporal context while keeping the storage and computation requirements low.”

Others are taking a more creative approach. Twenty Billion Neurons, a startup based in Toronto and Berlin, created a custom data set by paying crowdsourced workers to perform simple tasks. One of the company’s cofounders, Roland Memisevic, says it also uses a neural network designed specifically to process temporal vision information.

“Networks trained on the other data sets can tell you whether the video shows a soccer match or a party,” he says. “Our networks can tell you whether someone just entered the room.”

Danny Gutfreund, a researcher at IBM who collaborated on the project, says recognizing actions effectively will require that machines learn about, say, a person taking an action and transfer this knowledge to a case where, say, an animal is performing the same action. Progress in this area, known as transfer learning, will be important for the future of AI. “Let’s see how machines can do this transfer learning, this analogy, that we do very well,” he says.

Gutfreund adds that the technology could have practical applications. “You could use it for elder care, telling if someone has fallen or if they have taken their medicine,” he says. “You can think of devices that help blind people.”