MUSTI Challenge: Multimodal Understanding of Smells in Texts and Images

Image source: https://rkd.nl/explore/images/278792

Calling all NLP & Computer Vision researchers to take on our new challenge: Multimodal Understanding of Smells in Texts and Images !

In the context of the MediaEval Multimedia Evaluation benchmark, we have developed a task that brings together recognition of visual and textual information regarding smells. We have set up 2 subtasks:

  1. Classification: Predict whether a text passage and an image evoke the same smell source or not.
  2. Detection: identify the common smell source(s) between the text passages and images. Smell sources include persons, objects or places that have a specific smell, or that produce odors (e.g. plant, animal, perfume, human)

This task is driven by the following research questions:

  • What does it mean for a text passage and an image to be related in terms of smell?
  • Do different text and image genres reference smell differently?
  • Do different languages reference smell differently?
  • How do references to smell in texts and images change over time?
  • How do relationships between smell references in texts and images change over time?

Data & Evaluation
We have put together a set of copyright-free texts and images. It contains texts in English, German, Italian, and French that participants are to match to the images. The images are annotated with 80+ categories of smell objects and gestures such as flowers, food, animals, sniffing and holding the nose. The object categories are organised in a two-level taxonomy.

The Odeuropa text and image benchmark datasets are available as training data to the participants. The image dataset consists of ~3000 images with 20,000 associated object annotations and 600 gesture annotations.

Submissions will be evaluated on a held-out dataset of roughly 1,200 images with associated texts in the four languages. We will evaluate using multiple statistics as each provides a slightly different perspective on the results. Main Task: Predicting whether an image and a text passage evoke the same smell source or not

Important Dates

  • 23 November 2022: Results returned
  • 28 November 2022: Working notes paper
  • 12-13 January 2023: 13th Annual MediaEval Workshop, Collocated with MMM 2023 in Bergen, Norway & online.

More information and registration:

https://multimediaeval.github.io/editions/2022/tasks/musti/

 

 

Improve your olfactory language – Hackathon

Image source: https://en.wikipedia.org/wiki/File:National_Library_Ljubljana_2010.jpg

Join us on 7 & 8 November 2022 at the National and University Library of Slovenia in Ljubljana for a nose-dive into the history and heritage of smell and the Odeuropa tools and data!

For whom? 
The event is directed at cultural heritage professionals, digital heritage collection specialists, computer science scholars, historians, (computational) linguists.

Why participate?
The hackathon will offer an intensive, two-day, nose-on and hands-on meeting, to try out cutting edge research and development in sensory data-mining, and discuss the opportunities of olfactory approaches to cultural collections and archives. Also, the rich olfactory lexicons of the past may offer new opportunities for research and exploitation. The nose-on setup of the workshop, which includes sniffing sessions and a smell walk through the library, will offer participants a chance to learn by sensing. Furthermore, we offer a reception in the library, where you can make valuable new connections with an international network of colleagues.

What can you expect?
In the morning of 7 November, we will start with a general introduction to the topic of the history, heritage and smell language, accompanied by sniffing sessions. Researchers of the Odeuropa project will highlight the opportunities of working with smells in heritage institutes, and what we can learn from the rich smell language of the past. Furthermore, the models and demonstrators of the Odeuropa project will be presented. This workshop part of the hackathon is also open for the general public.

In the afternoon of 7 November and on 8 November the participants will work in small groups to test and evaluate the Odeuropa prototype tools on concrete tasks, such as:

  • Search and find the occurrence of odors in historical texts using the Odeuropa Explorer & Smell Tracker
  • Analyse the descriptions of olfactory events in different languages and domains
  • Try out automatic annotation of documents via Odeuropa Wikifier API
  • Study relatedness between Smell event, Smell Locations and Smell Appraisals / Emotions
  • Compare smell vocabularies in different languages
  • Trace odors in literary texts and explore the opportunities to ‘represent’ these scents to collection visitors

Furthermore, the participants are invited to bring their own datasets, or request specific tasks.

Smell walk:
The event also includes a smell walk (a walk where you are led by your nose) in and around the library: to train your nose, help to explore the concept of fragrant spaces, and discover the importance of the smell of heritage spaces.

Nose-Dive into the History and Heritage of Smell!

Practicalities
Participation is free, however, participants are expected to bring their own laptops, and cover their own travel expenses.

Apply until 25.10.2022:

https://docs.google.com/forms/d/e/1FAIpQLSewM1s1NGtDfp3NfDDlfJXosmJjhw18xf1xtYFkZ-O6vgjGWw/viewform?usp=sf_link

Contact email: lang-hackathon@ijs.si