Beauty Now in the Eye of the Algorithm

New technology from Xerox can sort photos not just by their content but also according to their aesthetic qualities, such as which portraits are close-in and well-lit, or which wildlife shots are least cluttered.

Still in the prototype stage, the technology could eventually help with tasks like choosing which of hundreds of digital photos taken on a family vacation should appear in a photo album. It could help stock agencies sort photos by their characteristics, and it could be deployed inside a camera to help people delete lower-quality scenes quickly, saving on storage space and hassle.

“What they show is that now you don’t need a human to select images that are going to be judged beautiful,” says Aude Oliva, an associate professor of brain and cognitive sciences at MIT, who also works on image recognition. “You can run the algorithm, and it will give a good estimate.”

The technology—developed at the Xerox Research Center Europe in Grenoble, France—is slated for beta testing with Xerox corporate partners next year, says Craig Saunders, manager of the computer vision research group there. These partners include graphic design firms, online photo-book companies, and stock agencies, all of which might want new ways to sort and find photos.

The Xerox system learns about quality photography by studying photos that had previously been chosen for public display in online photo albums, such as public ones shown on Facebook, or photos tagged as high quality on Flickr. Then it notes common characteristics of these photos.

Not surprisingly, these characteristics often correspond to what experts already understand about good photographs. The best portraits of people, for example, have indirect lighting and blurry or monochromatic backgrounds that help keep the focus on the person. Good beach photos often include silky-looking waves, a trick achieved through slow shutter speeds. And many kinds of photos are appealing because they follow the “rule of threes,” with subjects divided among three zones in the photo. “We try to learn what it is about these features that makes photos ‘good,’” says Saunders. (Examples and demonstrations can be found here.)

The technique builds on a larger body of research, conducted at Xerox and many other labs, that strives to improve image recognition by breaking down photos into what researchers call a visual vocabulary—corners and edges that might define buildings, round shapes that might be wheels, regions of green that might indicate landscapes, and many more such elements (and combinations thereof). The resulting technologies build up knowledge about what pieces correspond to certain types of images by examining Internet-based photos that are already tagged with text identifying what’s in them.

Many research groups, including the one at Xerox, are working on improving not only the accuracy of these methods but also their computational efficiency. For example, Xerox announced recently that it has developed a system capable of finding images that have similar characteristics. It can sort through five million images in less than a second.

Xerox plans to launch this tool next year as a cloud-based service that could be used to refine searches in large image repositories like stock photo agencies. The company also released a related Facebook app, called Catepix, that examines your Facebook photos, categorizes them (portrait, landscape, etc.), and tells you what they say about your personality.

Unfortunately, I have posted only three pictures on Facebook, so the app failed to tell me much of anything. But it did put up a post under my name declaring that I was a portrait kind of guy.