Google

Mood Board Search: Google ML tool that leverages mood boards as a query over image collections

Advances in computer vision and natural language processing continue to unlock new ways of exploring billions of images available on public and searchable websites. Google researchers have been working collaboratively with artists, photographers, and image researchers to explore how ML might enable people to use expressive queries as a way of visually exploring datasets.

Google recently launched an outcome of this partnership, Mood Board Search, a new ML-powered research web tool that leverages mood boards as a query over image collections. It is a web-based tool that lets you train a computer to recognize visual concepts using mood boards and machine learning. It’s a playful way to explore and analyze image collections using mood boards as your search query. With the help of this tool, users can independently define and evoke visual notions.

Mood Board Search:

The Google research team aimed to create a flexible and approachable interface for people without ML experience to train a computer to recognize a visual notion as they see it. It includes three visual concepts developed by artist collaborators that are FRACTURED by Rachel Maggart; SIGHTUNSEEN by Alexander Etchells; STATESOFMIND by Tom Hatton. Three artists created visual concepts to share their way of seeing, shown here in an experimental app by design invention studio, Nord Projects. The team is still in the developing phase of the research tool.

Mood Board Search can be used for ambiguous inquiries, such as “peaceful,” or for words and specific images that might not be exact enough to yield beneficial results in a regular search. With Mood Board Search, users can train a computer to recognize visual concepts in image collections. It’s also possible to signal which images are more important to a visual concept by up weighting or down weighting images, or by adding images that are the opposite of the concept.

Mood Board Search uses existing pre-trained computer vision models like GoogLeNet and MobileNet and a machine learning technique known as concept activation vectors (CAVs). Google uses CAVs to find a model’s sensitivity to a mood board created by the user. In other words, each mood board creates a CAV, a direction in embedding space. This study methodology was initially made publicly available by Google. This is the approach behind features like Focus mode and AI crop.

Future work will expand ML models and inputs to allow even deeper subjective findings, independent of medium, and learning about new types of human-machine collaboration utilizing mood board search. And Google plans to use Mood Board Search to learn about new forms of human-machine collaboration and expand ML models and inputs like text and audio, to allow even deeper subjective discoveries, regardless of medium.