The DEEP Sensorium: a multidimensional approach to sensory domain labelling
Authors: 
Simona Corciulo, Livio Bioglio, Valerio Basile, Viviana Patti, 
Rossana DamianoAuthors Info & Claims
Pages 661 - 668
Abstract
In this paper, we describe our intuitions about how language technologies can contribute to create new ways to enhance the accessibility of exhibits in cultural contexts by exploiting the knowledge about the history of our senses and the link between perception and language.
We evaluate the performance of five multi-class classification models for the task of sensory recognition and introduce the DEEP Sensorium (Deep Engaging Experiences and Practices - Sensorium), a multidimensional dataset that combines cognitive and affective features to inform systematic methodologies for augmenting exhibits with multi-sensory stimuli.
For each model, using different feature sets, we show that the features expressing the affective dimension of words combined with sub-lexical features perform better than uni-dimensional training sets.
References
[1]
Lawrence W. Barsalou and Katja Wiemer-Hastings. 2005. Situating Abstract Concepts. In Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking, Diane Pecher and Rolf A.Editors Zwaan (Eds.). Cambridge University Press, Cambridge, 129–163. https://doi.org/10.1017/CBO9780511499968.007
[2]
Mareike Bayer and Annekathrin Schacht. 2014. Event-related brain responses to emotional words, pictures, and faces–a cross-domain comparison. Frontiers in psychology 5 (2014), 1106.
[3]
Jeffrey R Binder, Chris F Westbury, Kristen A McKiernan, Edward T Possing, and David A Medler. 2005. Distinct brain systems for processing concrete and abstract concepts. Journal of cognitive neuroscience 17, 6 (2005), 905–917.
[4]
Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49–59.
[5]
Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. Journal of artificial intelligence research 49 (2014), 1–47.
[6]
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior research methods 46 (2014), 904–911.
[7]
Curt Burgess and Kay Livesay. 1998. The effect of corpus size in predicting reaction time in a basic word recognition task: Moving on from Kučera and Francis. Behavior Research Methods, Instruments, & Computers 30, 2 (1998), 272–277.
[8]
Max Coltheart. 1981. The MRC psycholinguistic database. The Quarterly Journal of Experimental Psychology Section A 33, 4 (1981), 497–505.
[9]
Radu Comes. 2016. Haptic devices and tactile experiences in museum exhibitions. Journal of Ancient History and Archaeology 3, 4 (2016), 60–64.
[10]
Simona Corciulo and Viviana Patti ad Damiano Rossana. 2022. Towards the Construction of a Dataset of Art-Related Synaesthetic Metaphors: Methods and Results. In Proceedings of the 1st Workshop on Artificial Intelligence for Cultural Heritage, AI4CH 2022, co-located with the 21st International Conference of the Italian Association for Artificial Intelligence (AIxIA 2022), Udine, Italy, November 28, 2022. CEUR Workshop Proceedings, Udine, Italy, 113–125. http://ceur-ws.org/Vol-3286/12_paper.pdf
[11]
Jasper HB De Groot, Gün R Semin, and Monique AM Smeets. 2014. I can see, hear, and smell your fear: comparing olfactory and audiovisual media in fear communication.Journal of Experimental Psychology: General 143, 2 (2014), 825.
[12]
Melissa Donaldson. 2017. Plutchik’s wheel of emotions—2017. Update.
[13]
Mohamad A Eid and Hussein Al Osman. 2015. Affective haptics: Current research and future directions. IEEE Access 4 (2015), 26–40.
[14]
Yoav Gilad, Victor Wiebe, Molly Przeworski, Doron Lancet, and Svante Pääbo. 2004. Loss of olfactory receptor genes coincides with the acquisition of full trichromatic vision in primates. PLoS biology 2, 1 (2004), e5.
[15]
Ian Grosvenor and Natasha Macnab. 2013. Seeing through touch: the material world of visually impaired children. Educar em Revista49 (2013), 39–57.
[16]
John F Hall. 1954. Learning as a function of word-frequency. The American journal of psychology 67, 1 (1954), 138–140.
[17]
Matthew J Hertenstein, Rachel Holmes, Margaret McCullough, and Dacher Keltner. 2009. The communication of emotion via touch.Emotion 9, 4 (2009), 566.
[18]
Matthew J Hertenstein, Dacher Keltner, Betsy App, Brittany A Bulleit, and Ariane R Jaskolka. 2006. Touch communicates distinct emotions.Emotion 6, 3 (2006), 528.
[19]
Paul Hoffman, Matthew A Lambon Ralph, and Timothy T Rogers. 2013. Semantic diversity: A measure of semantic ambiguity based on variability in the contextual usage of words. Behavior research methods 45 (2013), 718–730.
[20]
Martin Jay. 1993. 1. The Noblest of the Senses: Vision from Plato to Descartes. In Downcast Eyes. University of California Press, Berkeley, 21–82. https://doi.org/10.1525/9780520915381-003
[21]
Danijela Kambaskovic-Sawers and Charles T Wolfe. 2014. The senses in philosophy and science: from the nobility of sight to the materialism of touch.
[22]
Carolyn Korsmeyer. 2019. A Tour of the Senses. The British Journal of Aesthetics 59, 4 (2019), 357–371.
[23]
Gloria L Krahn. 2011. WHO World Report on Disability: a review. Disability and health journal 4, 3 (2011), 141–142.
[24]
George Lakoff. 2008. The neural theory of metaphor. In The Cambridge Handbook of Metaphor and Thought, Raymond W.Editor Gibbs, Jr. (Ed.). Cambridge University Press, Cambridge, 17–38. https://doi.org/10.1017/CBO9780511816802.003
[25]
Dermot Lynott and Louise Connell. 2009. Modality exclusivity norms for 423 object properties. Behavior Research Methods 41, 2 (2009), 558–564.
[26]
Dermot Lynott and Louise Connell. 2013. Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form. Behavior research methods 45, 2 (2013), 516–526.
[27]
Michael Macht, S Roth, and Heiner Ellgring. 2002. Chocolate eating in healthy men during experimentally induced sadness and joy. Appetite 39, 2 (2002), 147–158.
[28]
Asifa Majid, Seán G Roberts, Ludy Cilissen, Karen Emmorey, Brenda Nicodemus, Lucinda O’grady, Bencie Woll, Barbara LeLan, Hilário De Sousa, Brian L Cansler, 2018. Differential coding of perception in the world’s languages. Proceedings of the National Academy of Sciences 115, 45 (2018), 11369–11376.
[29]
M. McLuhan. 1962. The Gutenberg Galaxy: The Making of Typographic Man. University of Toronto Press, Toronto. https://books.google.it/books?id=y4C644zHCWgC
[30]
Stefano Menini, Teresa Paccosi, Serra Sinem Tekiroğlu, and Sara Tonelli. 2022. Building a Multilingual Taxonomy of Olfactory Terms with Timestamps. In Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 4030–4039. https://aclanthology.org/2022.lrec-1.429
[31]
Saif Mohammad. 2018. Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 174–184. https://doi.org/10.18653/v1/P18-1017
[32]
Saif Mohammad. 2018. Word Affect Intensities. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan, 174–183. https://aclanthology.org/L18-1027
[33]
Pilar Fatás Monforte. 2021. The Cave of Altamira and Modern Artistic Creation. In Visual Culture, Heritage and Identity: Using Rock Art to Reconnect Past and Present. Archaeopress Publishing Ltd, Oxford, UK, 24.
[34]
Konstantinos Moustakas, Michael G Strintzis, Dimitrios Tzovaras, Sébastien Carbini, Olivier Bernier, Jean-Emmanuel Viallet, Stephan Raidt, Matei Mancas, Mariella Dimiccoli, Enver Yagci, 2006. Masterpiece: physical interaction and 3D content-based search in VR applications. IEEE MultiMedia 13, 3 (2006), 92–100.
[35]
Yen Nguyen and Charles N Noussair. 2014. Risk aversion and emotions. Pacific economic review 19, 3 (2014), 296–312.
[36]
Andrew Ortony and Lynn Fainsilber. 1987. The role of metaphors in descriptions of emotions. In Theoretical Issues in Natural Language Processing 3.
[37]
Charles Egerton Osgood, George J Suci, and Percy H Tannenbaum. 1957. The measurement of meaning. University of Illinois press, Champaign, Illinois, US.
[38]
Allan Paivio. 1991. Dual coding theory: Retrospect and current status.Canadian Journal of Psychology/Revue canadienne de psychologie 45, 3 (1991), 255.
[39]
Allan Paivio, John C Yuille, and Stephen A Madigan. 1968. Concreteness, imagery, and meaningfulness values for 925 nouns.Journal of experimental psychology 76, 1p2 (1968), 1.
[40]
Marina Palazova, Werner Sommer, and Annekathrin Schacht. 2013. Interplay of emotional valence and concreteness in word processing: An event-related potential study with verbs. Brain and language 125, 3 (2013), 264–271.
[41]
Sophie Pauligk, Sonja A Kotz, and Philipp Kanske. 2019. Differential impact of emotion on semantic processing of abstract and concrete words: ERP and fMRI evidence. Scientific Reports 9, 1 (2019), 14439.
[42]
Sarah Pink. 2006. The future of visual anthropology: Engaging the senses. Routledge, London, UK.
[43]
Tom Pursey and David Lomas. 2018. Tate Sensorium: An experiment in multisensory immersive design. The Senses and Society 13, 3 (2018), 354–366.
[44]
Stanley Rachman. 1980. Emotional processing. Behaviour research and therapy 18, 1 (1980), 51–60.
[45]
Mahendra Sahare and Hitesh Gupta. 2012. A review of multi-class classification for imbalanced data. International Journal of Advanced Computer Research 2, 3 (2012), 160.
[46]
Paula J Schwanenflugel. 2013. Why are abstract concepts hard to understand? In The psychology of word meanings. Psychology Press, London, UK, 235–262.
[47]
Paula J Schwanenflugel and Edward J Shoben. 1983. Differential context effects in the comprehension of abstract and concrete verbal materials.Journal of Experimental Psychology: Learning, Memory, and Cognition 9, 1 (1983), 82.
[48]
Bernard J Shapiro. 1969. The subjective estimation of relative word frequency. Journal of verbal learning and verbal behavior 8, 2 (1969), 248–251.
[49]
Chang Su, Xiaomei Wang, Zita Wang, and Yijiang Chen. 2019. A model of synesthetic metaphor interpretation based on cross-modality similarity. Computer Speech & Language 58 (2019), 1–16.
[50]
Robert W Sussman, D Tab Rasmussen, and Peter H Raven. 2013. Rethinking primate origins again. American Journal of Primatology 75, 2 (2013), 95–106.
[51]
Serra Sinem Tekiroğlu, Gözde Özbal, and Carlo Strapparava. 2014. Sensicon: An automatically constructed sensorial lexicon. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1511–1521.
[52]
Serra Sinem Tekiroğlu, Gözde Özbal, and Carlo Strapparava. 2015. Exploring sensorial features for metaphor identification. In Proceedings of the Third Workshop on Metaphor in NLP. Association for Computational Linguistics, Denver, Colorado, 31–39. https://aclanthology.org/W15-1404
[53]
Gyanendra K Verma and Uma Shanker Tiwary. 2014. Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals. NeuroImage 102 (2014), 162–172.
[54]
Gabriella Vigliocco, Lotte Meteyard, Mark Andrews, and Stavroula Kousta. 2009. Toward a theory of semantic representation. Language and Cognition 1, 2 (2009), 219–247.
[55]
Caterina Villani, Luisa Lugli, Marco Tullio Liuzza, Roberto Nicoletti, and Anna M Borghi. 2021. Sensorimotor and interoceptive dimensions in concrete and abstract concepts. Journal of memory and language 116 (2021), 104173.
[56]
Bodo Winter. 2016. The sensory structure of the English lexicon. University of California, Merced.
[57]
Bodo Winter. 2016. The sensory structure of the English lexicon. University of California, Merced, Merced, US.
[58]
Jiyun Yang and Jeehyun Lee. 2019. Application of sensory descriptive analysis and consumer studies to investigate traditional and authentic foods: A review. Foods 8, 2 (2019), 54.
[59]
Lingyao Yuan and Jordan Barlow. 2021. Sensitive to the Digital Touch? Exploring Sensory Processing Sensitivity and Its Impact on Anthropomorphized Products in E-Commerce. In 54th Hawaii International Conference on System Sciences, HICSS 2021, Kauai, Hawaii, USA, January 5, 2021. ScholarSpace, Kauai, Hawaii, US, 1–10.
[60]
Jonathan R Zadra and Gerald L Clore. 2011. Emotion and perception: The role of affective information. Wiley interdisciplinary reviews: cognitive science 2, 6 (2011), 676–685.
[61]
Jie Zhou, Qi Su, and Pengyuan Liu. 2020. A metaphorical analysis of five senses and emotions in mandarin Chinese. In Chinese Lexical Semantics: 20th Workshop, CLSW 2019, Beijing, China, June 28–30, 2019, Revised Selected Papers 20. Springer, 607–617.
Index Terms
- The DEEP Sensorium: a multidimensional approach to sensory domain labelling
Recommendations
Attention and Sensory Interactions within the Occipital Cortex in the Early Blind: An fMRI Study
Visual deprivation early in life results in occipital cortical responsiveness across a broad range of perceptual and cognitive tasks. In the reorganized occipital cortex of early blind (EB) individuals, the relative lack of specificity for particular ...
Tickling Expectations: Neural Processing in Anticipation of a Sensory Stimulus
Predictions of the near future can optimize the accuracy and speed of sensory processing as well as of behavioral responses. Previous experience and contextual cues are essential elements in the generation of a subjective prediction. Using a blocked ...
Interactions between voluntary and stimulus-driven spatial attention mechanisms across sensory modalities
In everyday life, the allocation of spatial attention typically entails the interplay between voluntary (endogenous) and stimulus-driven (exogenous) attention. Furthermore, stimuli in different sensory modalities can jointly influence the direction of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2023
1567 pages
ISBN:9781450394192
DOI:10.1145/3543873
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 April 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
WWW '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 269Total Downloads
- Downloads (Last 12 months)149
- Downloads (Last 6 weeks)26
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in