Revealing the True Colors of Masterworks

Enhancements to image-processing technologies for colorizing black-and-white images are helping curators divine the colors used by the French artist Henri Matisse on his landmark work Bathers by a River–while the painting was still a work in progress

The tricks deployed by curators could be more widely relevant to other colorizing applications where it’s not obvious what the colors should be in a black-and-white image of a piece of art, or in cases where subtle differences are important and should be highlighted, such as in medical images.

Researchers at Northwestern University used information about Matisse’s prior works, as well as color information from test samples of the work itself, to help colorize a 1913 black-and-white photo of the work in progress. Matisse began work on Bathers in 1909 and unveiled the painting in 1917.

In this way, they learned what the work looked like midway through its completion. “Matisse tamped down earlier layers of pinks, greens, and blues into a somber palette of mottled grays punctuated with some pinks and greens,” says Sotirios A. Tsaftaris, a professor of electrical engineering and computer science at Northwestern. That insight helps support research that Matisse began the work as an upbeat pastoral piece but changed it to reflect the graver national mood brought on by World War I.

The process was more complex than the methods used routinely for colorizing old movies and family photographs. In those kinds of applications, backgrounds such as skies, clothing, and skin tones are “more homogeneous and thus easier to extrapolate,” says Tsaftaris. The color of an entire sky can be determined from a relatively small batch of pixel data, he said. It’s far harder in a black-and-white image of a piece of color art, because “the painter works from a very unique palette of colors that is particular to him, that he sees in his mind,” adds Aggelos Katsaggelos, a professor of electrical engineering and computer science at Northwestern, who collaborated with Tsaftaris.

The researchers made a high-resolution digital version of the 1913 photograph to work from. The photograph itself contained crucial clues to colors and their saturation levels. But to draw a more complete picture, the scientists and their collaborators needed more data.

They took multiple digital photos of Bathers in its current form, going quadrant by quadrant to obtain a resolution of 4,000-by-5,000 pixels. Finally, they included information from historical accounts of what the painting looked like in 1909 and again in 1913, drawing on research by curators at the Art Institute of Chicago.

Finally, they used some sample data from their collaborators at the Art Institute of Chicago: cross-sections of the hidden paint layers on Bathers, obtained by removing microscopic core samples of the painting for spectroscopic analysis.

They applied all of this information to help colorize the photograph, taken by photographer Eugene Druet in November 1913.

And when all of the data sources were combined, it allowed the researchers to transfer colors onto their digital photo of the old photo. The algorithm’s job at that point was to propagate the transferred colors across the entire digital photo, pixel by pixel, to rediscover some of the painting’s 1913 appearance.

“This research is an excellent example of collaborative research between computer science, art conservation, and art history,” says Roy S. Berns, a chemist and color scientist at the Rochester Institute of Technology. “The historians bring their connoisseurship of the artist and their oeuvre. The conservators contribute their knowledge of artist materials and the artist’s working method. The computer scientists facilitate the visualization in a physically realistic way. Because the physical data are sparse, collaboration is required to ensure the result is plausible.”

The effort took three years, and the scientists and conservators say they held back their findings until they reached a 95 percent confidence level about the colorized image.

The algorithm can be tweaked to work with other similar situations and other artists. While this algorithm was “customized to work on paintings and on the particular style of Matisse,” Tsaftaris said, “we can turn off some options, and it works on other paintings as well.”

Tsaftaris sees future applications of custom colorization, particularly in the medical field. The scientists are considering using their new methods to pseudo-colorize grayscale cardiac magnetic resonance images (MRI) to make it easier for doctors to read, analyze, and render a diagnosis. In this case, they might use cues gleaned from color images of diseased hearts, for example, to inform their work in how to properly colorize black-and-white MRI images to bring out the most relevant distinctions.