MUSTI Challenge: Multimodal Understanding of Smells in Texts and Images

Image source:

Calling all NLP & Computer Vision researchers to take on our new challenge: Multimodal Understanding of Smells in Texts and Images !

In the context of the MediaEval Multimedia Evaluation benchmark, we have developed a task that brings together recognition of visual and textual information regarding smells. We have set up 2 subtasks:

  1. Classification: Predict whether a text passage and an image evoke the same smell source or not.
  2. Detection: identify the common smell source(s) between the text passages and images. Smell sources include persons, objects or places that have a specific smell, or that produce odors (e.g. plant, animal, perfume, human)

This task is driven by the following research questions:

  • What does it mean for a text passage and an image to be related in terms of smell?
  • Do different text and image genres reference smell differently?
  • Do different languages reference smell differently?
  • How do references to smell in texts and images change over time?
  • How do relationships between smell references in texts and images change over time?

Data & Evaluation
We have put together a set of copyright-free texts and images. It contains texts in English, German, Italian, and French that participants are to match to the images. The images are annotated with 80+ categories of smell objects and gestures such as flowers, food, animals, sniffing and holding the nose. The object categories are organised in a two-level taxonomy.

The Odeuropa text and image benchmark datasets are available as training data to the participants. The image dataset consists of ~3000 images with 20,000 associated object annotations and 600 gesture annotations.

Submissions will be evaluated on a held-out dataset of roughly 1,200 images with associated texts in the four languages. We will evaluate using multiple statistics as each provides a slightly different perspective on the results. Main Task: Predicting whether an image and a text passage evoke the same smell source or not

Important Dates

  • 23 November 2022: Results returned
  • 28 November 2022: Working notes paper
  • 12-13 January 2023: 13th Annual MediaEval Workshop, Collocated with MMM 2023 in Bergen, Norway & online.

More information and registration: