Calling all NLP & Computer Vision researchers to take on our new challenge: Multimodal Understanding of Smells in Texts and Images !
In the context of the MediaEval Multimedia Evaluation benchmark, we have developed a task that brings together recognition of visual and textual information regarding smells. We have set up 2 subtasks:
- Classification: Predict whether a text passage and an image evoke the same smell source or not.
- Detection: identify the common smell source(s) between the text passages and images. Smell sources include persons, objects or places that have a specific smell, or that produce odors (e.g. plant, animal, perfume, human)
This task is driven by the following research questions:
- What does it mean for a text passage and an image to be related in terms of smell?
- Do different text and image genres reference smell differently?
- Do different languages reference smell differently?
- How do references to smell in texts and images change over time?
- How do relationships between smell references in texts and images change over time?
Data & Evaluation
We have put together a set of copyright-free texts and images. It contains texts in English, German, Italian, and French that participants are to match to the images. The images are annotated with 80+ categories of smell objects and gestures such as flowers, food, animals, sniffing and holding the nose. The object categories are organised in a two-level taxonomy.
The Odeuropa text and image benchmark datasets are available as training data to the participants. The image dataset consists of ~3000 images with 20,000 associated object annotations and 600 gesture annotations.
Submissions will be evaluated on a held-out dataset of roughly 1,200 images with associated texts in the four languages. We will evaluate using multiple statistics as each provides a slightly different perspective on the results. Main Task: Predicting whether an image and a text passage evoke the same smell source or not
Important Dates
- 23 November 2022: Results returned
- 28 November 2022: Working notes paper
- 12-13 January 2023: 13th Annual MediaEval Workshop, Collocated with MMM 2023 in Bergen, Norway & online.
More information and registration:
https://multimediaeval.github.io/editions/2022/tasks/musti/
