Finding references to smell in artworks

Identifying visual references of olfactory phenomena in artworks is an important way to uncover how Europe may have smelled in the past and how smell was represented. The computer-vision team of the Odeuropa project is currently working on methods which would automatically extract these references from various large collections of European artworks by applying, modifying, and extending state-of-the-art object detection methods. In order to collect and extract these olfactory references using computer recognition, it is necessary to first identify how smell is visually represented or depicted in historical artworks.

To provide an example of how this works, we used the print Smell (1581-1656) by Nicolaes de Bruyn, which is currently housed at the Rijksmuseum in Amsterdam.

Smell (1581-1656) by Nicolaes de Bruyn

In the sixteenth century, the pairing of a woman with a dog was used as a visual depiction or personification of the sense of smell. Since the object detection method was able to identify the dog and the woman, this would seem like an effective system. However, there are certain challenges which come with this detection. Firstly, not all pairings of people or women with dogs are ‘olfactory’, for example in other centuries a dog on the lap or feet of a woman represents fidelity, as seen in Jan van Eyck’s Arnolfini Portrait (1434).

Jan van Eyck’s Arnolfini Portrait (1434)
This presents us with the challenge of distinguishing when a dog is or is not ‘olfactory’ in nature. A second challenge is that the olfactory gesture of the woman smelling the flowers was also not detected by computer recognition. This poses further limitations on detecting olfactory elements in paintings.

Many olfactory-related narratives can also be found in the Bible, the Sacrifice of Noah (Genesis 8:20) for example. The print, Sacrifice of Noah after the Flood by Casper Luyken, shows Noah creating a burnt offering of animals, combined with the usual “Covenant of the Rainbow” in the background.

Sacrifice of Noah after the Flood by Casper Luyken

These types of olfactory narratives reveal more limitations of existing object detectors, while the people and animals were easily detected but the rainbow and cloud of smoke were not, hence overlooking the olfactory element of the artwork. This could be because these object detection systems are limited to the data with which they have been trained, leading to two problems. Firstly, since the detectors are trained with photographic data, their effectiveness decreases when applied to images with an artistic style such as historical paintings and prints. Secondly, it could be that certain objects (like smoke and rainbows) were either underrepresented or not at all part of the detector’s training data.

In order to tackle these issues of computer recognition, we will apply and modify domain adaptation techniques in order to improve the detection abilities on artistic image domains. After implementing a working object detection system, we plan to incorporate art historical knowledge which would also enable our system to recognize complex and context-specific olfactory references.

Submit your work: First International Workshop on Multisensory Data & Knowledge

Together with the Polifonia team, we’re organising a workshop on Multisensory Data & Knowledge to take place in conjunction with the Language Data and Knowledge conference in September. The goal of this workshop is to advance our understanding of how smells and music are represented in texts and structured data. The topics we want to address revolve around extracting references to smells, music, context, and visual information from text as well as relevant data describing their cultural, historical and political context, and model them in the form of interlinked knowledge graphs. This research has a strong interdisciplinary character, hence the workshop has the potential to attract researchers from diverse disciplines from both social sciences and humanities and computer science. Its potential impact is significant to many application areas including: preservation and valorisation of cultural heritage, data-driven policy making in cultural heritage, urban planning, artistic performances, applications for scholars in musicology and history, applications for museums, innovation in teaching, maintenance and exploitation of large catalogues, archives and libraries.

We invite long papers between 10 and 15 pages and short papers between 6 to 8 pages. Note that this workshop is organised following the computer science conference/publication culture, so initial submissions are expected to be in a near publishable state and will be reviewed by three reviewers. Accepted papers will be published through ceur-ws.org.

Submission deadline: 23 April 2021, the workshop will take place on 1 September.

More information on the workshop: https://odeuropa.github.io/mdk21/

Opening the fragrant conversation

When we announced Odeuropa last November, we couldn’t have dreamed that it would be received with so much enthusiasm. As the research teams kick off the programme, we’d like to share an overview of the international response to Odeuropa’s launch from the media, research community and general public, which is already developing into a wider conversation into the relevance of smell in our lives and how the idea of olfactory heritage resonates with a large variety of audiences.

Not only in Europe, but around the globe, the project was widely covered by feature articles and in-depth interviews, for instance by the Spanish scientific news agency,  Le Monde, La Stampa, La RepubblicaDeutschland Radio, NPO Radio 1, Delo, El PaísCNET, and CNN. The Guardian, The New York Times and The Sunday Times reported on our research aims and innovative approach, along with the BBC.

From a video by Indonesian YouTube channel The Shiny Peanut (+12M subscribers) which animated members of our research team to a thoughtful reflection by the Times of Israel of the power of scent to overcome ignorance and communicate on a subject as complex as the Holocaust, we were delighted to see that so many people across the world share our curiosity to explore the potential of olfactory heritage.

Shiny Peanut
The Shiny Peanut, YouTube.

People shared the scents they considered meaningful in comments about nostalgic memories or fragrant experiences, and even reflections about covid-related loss of smell.  The New York Times, inspired by Odeuropa, invited their readers to imagine a museum of smells and worked with artist Janie Korn to produce a series of one-off candles inspired by the project

We are honoured and intensely happy to engage with these and the many other responses we received. In the next weeks and months, we will start to explore how we can accommodate new collaborations and keep expanding our network. Please stay tuned for information here on our website and via Twitter @odeuropa