MIT Algorithm Discovers Subtle Links Between Great Works of Art With AI

A new algorithm from MIT researchers called "MosAIc" is discovering interesting yet deeply crucial similarities between works of art on display at major museums in New York City and Amsterdam, according to a blog post shared on MIT's Computer Science and Artificial Intelligence Lab.

This kind of AI could even help us express the limits of generative adversarial networks (GAN).

RELATED: DEEP LEARNING REACHING COMPUTATIONAL LIMITS, WARNS NEW MIT STUDY

New AI system finds subtle links between great artworks

The new MIT-built system is uncovering unnoticed similarities between works of art on display at New York's Metropolitan Museum of Art (MET) and Amsterdam's Rijksmuseum. MosAIc scans an image, then makes use of deep networks to find similarities between different artworks across disparate cultures, media, and artists — undiscovered until now, reports Engadget.

In one example, MosAIc found a link between Jan Asselijn's "The Threatened Swan" and Francisco de Zurbarán's "The Martyrdom of Saint Serapion." An MIT CSAIL doctoral student named Mark Hamilton who was lead author on a paper regarding the MosAIc project said: "These two artists did not have correspondence or meet each other during their lives, yet their paintings hinted at a rich, latent structure that underlies both of their works."

Microsoft, MIT design MosAIC system

Microsoft joined MIT's CSAIL Laboratory in the design of MoSAIc — which took inspiration from the "Rembrandt and Velazquez" exhibit in the Rijksmuseum, curated to pair up paintings that might look different yet share a deeper connection through interpretative or critical styles, The Next Web reports.

Researchers enter a query like "which musical instrument is closest to this painting of a blue and white dress?" For this example, the algorithm replied with a blue and porcelain violin that helped researchers draw cultural exchanges between the Chinese and the Dutch.

AI built to match color, style, meaning, theme

In some respects, MosAIc isn't unlike Google's X degrees of separation experiment — which drew links between two images or artworks through a series of paintings. But MosAIc tops this because it only needs one image to find similar stylistic designs in other images. The new algorithm uses input images to match works across varying cultures.

Building the algorithm was challenging because the goal of matching images had to work based on not only similar color and style, but also theme and meaning, according to Hamilton. The full paper on the algorithm is available here.

AI art comparisons could explore limits of GAN

He and his colleagues used a novel K-Nearest Neighbor (KNN) data structure that links similar images via a tree-like figure, and they moved through the aesthetic structure until they discovered the closest result. Then they applied the algorithm to the combined open-access artworks of the Rijksmuseum and the MET.

The researchers also discovered that this AI method can help scientists find the limits of GAN (Generative Adversarial Network) on the basis of deepfake algorithms — and where they fail. However, it's still unclear if the algorithm may help distinguish deepfakes from the genuine artistic article.