News

MUSTI Challenge: Multimodal Understanding of Smells in Texts and Images

Image source: https://rkd.nl/explore/images/278792

Calling all NLP & Computer Vision researchers to take on our new challenge: Multimodal Understanding of Smells in Texts and Images !

In the context of the MediaEval Multimedia Evaluation benchmark, we have developed a task that brings together recognition of visual and textual information regarding smells. We have set up 2 subtasks:

  1. Classification: Predict whether a text passage and an image evoke the same smell source or not.
  2. Detection: identify the common smell source(s) between the text passages and images. Smell sources include persons, objects or places that have a specific smell, or that produce odors (e.g. plant, animal, perfume, human)

This task is driven by the following research questions:

  • What does it mean for a text passage and an image to be related in terms of smell?
  • Do different text and image genres reference smell differently?
  • Do different languages reference smell differently?
  • How do references to smell in texts and images change over time?
  • How do relationships between smell references in texts and images change over time?

Data & Evaluation
We have put together a set of copyright-free texts and images. It contains texts in English, German, Italian, and French that participants are to match to the images. The images are annotated with 80+ categories of smell objects and gestures such as flowers, food, animals, sniffing and holding the nose. The object categories are organised in a two-level taxonomy.

The Odeuropa text and image benchmark datasets are available as training data to the participants. The image dataset consists of ~3000 images with 20,000 associated object annotations and 600 gesture annotations.

Submissions will be evaluated on a held-out dataset of roughly 1,200 images with associated texts in the four languages. We will evaluate using multiple statistics as each provides a slightly different perspective on the results. Main Task: Predicting whether an image and a text passage evoke the same smell source or not

Important Dates

  • 23 November 2022: Results returned
  • 28 November 2022: Working notes paper
  • 12-13 January 2023: 13th Annual MediaEval Workshop, Collocated with MMM 2023 in Bergen, Norway & online.

More information and registration:

https://multimediaeval.github.io/editions/2022/tasks/musti/

 

 

The Toko’s Smell and Image

Written by: Josephine Koopman

Inspired by my master’s thesis on (post)colonial smellscapes, the idea arose to create a scent inspired by the smells of the toko. The scent features as the centerpiece of a small exhibition on the history of tokos. Alongside the scent, the exhibition features a selection of historical photographs depicting tokos in the former Dutch East Indies and the Netherlands. The exhibition is on display at the Floriade Expo 2022 until 9 October 2022. 

Last year, I joined Odeuropa as an intern to investigate (post)colonial smellscapes for the purpose of researching the relationship between smell and heritage, which resulted in the thesis (Post)colonial smellscapes in text and toko: an inquiry into olfactory heritage.

Colonial history is a particularly fragrant history. The sense of smell initiated colonial ventures as European expansion sprang from the desire for spices and other aromatic commodities. In the Netherlands today, our nose encounters traces of this history in ‘tokos’, grocery shops that offer food products from Asia, and sometimes also from Suriname. Its origins are rooted in the Dutch colonial past, roots which can be gleaned from the origin of the word itself. ‘Toko’ is Malay for ‘bazaar’. The ‘toko’ as a phenomenon entered Dutch culture from the East Indies in the early twentieth century.

Spices and seasonings at Toko Dun Yong, Amsterdam. Photo: Josephine Koopman.

My research into the smellscape of tokos played out in two parts. Firstly, I led several smellwalks that visited a few tokos around the Nieuwmarkt, in the old Chinatown of Amsterdam. The smellwalks were intended to uncover the smellscape of tokos answering the questions: what do tokos smell like, and what does its smellscape evoke in the smeller? Secondly, I conducted several interviews with toko shop owners and customers to find out their interpretations and evaluations of the tokos smellscape. In my thesis I propose that we can consider the smellscape of the toko as a ‘lieu de mémoire’ or a ‘site of memory’.

Smell research in times of covid-19. Fellow student Manthos in Toko Dun Yong, Amsterdam. Photo: Josephine Koopman

Presenting a Smellscape as Heritage

Through Odeuropa, I got in touch with Flevo Campus, a research institute based in Almere that is concerned with the future of food. Their project ‘Toko van de toekomst’ (‘Toko of the future’) explores how food-related knowledge embedded in communities is reproduced throughout generations. Flevo Campus asked me to create an exhibition about tokos for their location in the Floriade Expo 2022, an international horticulture event which takes place in Almere from 14 April to 9 October 2022.

The exhibition ‘Toko’s smell and image’ at Floriade Expo 2022, Almere. Photo: Josephine Koopman

The exhibition narrates the history of tokos through photography and smell. The centerpiece is a smell station which contains a scent inspired by the smells of the toko. The scent has been developed by Jorg Hempenius from Iscent specifically for this exhibition. On the basis of the accounts I collected over the course of the participatory smell research in tokos, Jorg created a scent that is layered and eclectic, reflecting the multiplicity of smells in the toko.  It is warm, salty and a bit funky: a celebratory clamor of spices and foods. 

Read more