More Web Proxy on the site http://driver.im/

research-article

Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance

Authors:

Ioannis Arapakis,

Ioannis Konstas,

Joemon M. JoseAuthors Info & Claims

MM '09: Proceedings of the 17th ACM international conference on Multimedia

Pages 461 - 470

https://doi.org/10.1145/1631272.1631336

Published: 19 October 2009 Publication History

Abstract

Multimedia search systems face a number of challenges, emanating mainly from the semantic gap problem. Implicit feedback is considered a useful technique in addressing many of the semantic-related issues. By analysing implicit feedback information search systems can tailor the search criteria to address more effectively users' information needs. In this paper we examine whether we could employ affective feedback as an implicit source of evidence, through the aggregation of information from various sensory channels. These channels range between facial expressions to neuro-physiological signals and are regarded as indicative of the user's affective states. The end-goal is to model user affective responses and predict with reasonable accuracy the topical relevance of information items without the help of explicit judgements. For modelling relevance we extract a set of features from the acquired signals and apply different classification techniques, such as Support Vector Machines and K-Nearest Neighbours. The results of our evaluation suggest that the prediction of topical relevance, using the above approach, is feasible and, to a certain extent, implicit feedback models can benefit from incorporating such affective features.

References

[1]

E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 19--26. ACM, 2006.

Digital Library

[2]

D. Andre, R. Pelletier, J. Farringdon, S. Safier, W. Talbott, R. Stone, N. Vyas, J. Trimble, D. Wolf, S. Vishnubhatla, S. Boehmke, J. Stivoric, and A. Teller. The development of the sensewear R armband, a revolutionary energy assessment device to assess physical activity and lifestyle. Technical report, BodyMedia, Inc., 2006.

[3]

I. Arapakis, J. M. Jose, and P. D. Gray. Affective feedback: an investigation into the role of emotions in the information seeking process. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 395--402. ACM, 2008.

Digital Library

[4]

I. Arapakis, Y. Moshfeghi, H. Joho, R. Ren, D. Hannah, and J. M. Jose. Enriching user profiling with affective features for the improvement of a multimodal recommender system. In Conference on Image and Video Retrieval, 2009.

Digital Library

[5]

R. Badi, S. Bae, J. M. Moore, K. Meintanis, A. Zacchi, H. Hsieh, F. Shipman, and C. C. Marshall. Recognizing user interest and document value from reading and organizing activities in document triage. In Proceedings of the 11th international conference on Intel ligent User Interfaces, pages 218--225, New York, NY, USA, 2006. ACM.

Digital Library

[6]

P. Bailey, N. Craswell, and D. Hawking. Engineering a multi-purpose test collection for web retrieval experiments. Inf. Process. Manage., 39(6):853--871, 2003.

Digital Library

[7]

N. J. Belkin. Anomalous state of knowledge for information retrieval. Canadian Journal of Information Science, 5:133--143, 1980.

[8]

D. Bilal and J. Kirby. Differences and similarities in information seeking: children and adults as web users. Information Processing and Management: an International Journal, 38(5):649--670, 2002.

Digital Library

[9]

W. Boucsein. Electrodermal activity. Plenum Press, New York, 1992.

[10]

C. H. Chan and G. J. F. Jones. Affect-based indexing and retrieval of films. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 427--430, New York, NY, USA, 2005. ACM.

Digital Library

[11]

G. Chanel, C. Rebetez, M. Betrancourt, and T. Pun. Boredom, engagement and anxiety as indicators for adaptation to difficulty in games. In MindTrek '08: Proceedings of the 12th international conference on Entertainment and media in the ubiquitous era, pages 13--17, New York, NY, USA, 2008. ACM.

Digital Library

[12]

R. R. Cornelius. Theoretical approaches to emotion. Proc ISCA: workshop on Speech and Emotion, pages 3--11, 2000.

[13]

W. Daelemans and A. van den Bosch. Memory-Based Language Processing (Studies in Natural Language Processing). Cambridge University Press, New York, NY, USA, 2005.

Digital Library

[14]

A. R. Damasio. Descartes Error: Emotion, Reason, and the Human Brain. Putnam/Grosset Press, 1994.

[15]

P. Ekman. Facial Expressions. Handbook of Cognition and Emotion. John Wiley&Sons Ltd., New York, 1999.

[16]

P. Ekman. Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. Times Books, New York, 2003.

[17]

P. Ekman. Unmasking the face. MA: Malor books, Cambridge, 2003.

[18]

P. Ekman and H. Oster. Facial expressions of emotion. Annual Review of Psychology, 30(1):527--554, 1979.

[19]

A. Hanjalic and L.-Q. Xu. Affective video content representation and modeling. Multimedia, IEEE Transactions on, 7(1):143--154, 2005.

Digital Library

[20]

F. Hopfgartner. A news video retrieval framework for the study of implicit relevance feedback. In Proceedings of the Second International Workshop on Semantic Media Adaptation and Personalization, pages 233--236, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[21]

http://www.polarusa.com.

[22]

H. Kataoka, H. Kano, H. Yoshida, A. Saijo, M. Yasuda, and M. Osumi. Delelopment of a skin temperature measuring system for non-contact stress evaluation. In Proceedings of the 20th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, volume 2, pages 940--943, 1998.

[23]

J. Koenemann and N. J. Belkin. A case for interaction: a study of interactive information retrieval behavior and effectiveness. In CHI '96: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 205--212, New York, NY, USA, 1996. ACM.

Digital Library

[24]

J. Kracker. Research anxiety and students' perceptions of research: an experiment. part i: Effect of teaching kuhlthau's isp model. Journal of the American Society for Information Science and Technology, 53(4):282--294, 2002.

Digital Library

[25]

I. Lopatovska and B. Mokros, H. Willingness to pay and experienced utility as measures of affective value of information ob jects: Users' accounts. Information Processing and Management: an International Journal, 44(1):92--104, 2008.

Digital Library

[26]

M. Lynne, R. C. Sheldrick, and R. Paulette. Affective dimensions of information seeking in the context of reading. Medford, NJ: Information Today, 2007.

[27]

C. Mooney, M. Scully, G. J. Jones, and A. F. Smeaton. Investigating Biometric Response for Information Retrieval Application, volume 3936/2006 of Lecture Notes in Computer Science. Springer Berlin, 2006.

Digital Library

[28]

D. Nahl. Learning the internet and the structure of information behavior. Journal of the American Society for Information Science, 49(11):1017--1023, 1998.

Digital Library

[29]

D. Nahl. Measuring the affective information environment of web searchers. In Proceedings of the American Society for Information Science and Technology, volume 41, pages 191--197, 2004.

[30]

D. Nahl and C. Tenopir. Affective and cognitive searching behavior of novice end-users of a full-text database. Journal of the American Society for Information Science, 47(4):276--286, 1996.

Digital Library

[31]

M. Pantic and L. Rothkrantz. Expert system for automatic analysis of facial expression. Image and Vision Computing Journal, 18(11):881--905, July 2000.

[32]

M. Pantic and L. J. M. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. In Proceedings of the IEEE, pages 1370--1390, 2003.

[33]

M. Pantic, N. Sebe, J. F. Cohn, and T. Huang. Affective multimodal human-computer interaction. In MULTIMEDIA '05: Proceedings of the 13th annual ACM international conference on Multimedia, pages 669--676, New York, NY, USA, 2005. ACM.

Digital Library

[34]

H.-R. Pfister and G. B¨ohm. The multiplicity of emotions: A framework of emotional functions in decision making. Judgment and Decision Making, 3:5--17, 2008.

[35]

R. W. Picard, A. Wexelblat, and C. I. N. I. Clifford I. Nass. Future interfaces: social and emotional. In CHI '02 extended abstracts on Human factors in computing systems, pages 698--699, New York, NY, USA, 2002. ACM.

Digital Library

[36]

K. Puolamaki, J. Salo jarvi, E. Savia, J. Simola, and S. Kaski. Combining eye movements and collaborative filtering for proactive information retrieval. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 146--153, New York, NY, USA, 2005. ACM.

Digital Library

[37]

B. Reeves and C. Nass. The media equation: how people treat computers, television, and new media like real people and places. Cambridge University Press, New York, NY, USA, 1996.

Digital Library

[38]

Y. Rui and S. Huang, T. Optimizing learning in image retrieval. In IEEE Proceedings of Conference on Computer Vision, volume 1, pages 236--243, 2000.

[39]

A. Russell, J., J. Bachorowski, and J. Fernandez-Dols. Facial and vocal expressions of emotion. Annual Review of Psychology, 2003.

[40]

J. A. Russell and J. H. Steiger. The Structure in Person's Implicit Taxonomy of Emotions. Journal of Research in Personality, 1982.

[41]

J. Salo jarvi, K. Puolam¨aki, and S. Kaski. Implicit Relevance Feedback from Eye Movements, volume Volume 3696/2005 of Artificial Neural Networks: Biological Inspirations - ICANN 2005. Springer Berlin, 2005.

Digital Library

[42]

T. Saracevic. Relevance reconsidered. information science: Integration in perspectives. Proceedings of the Second Conference on Conceptions of Library and Information Science, Copenhagen, Denmark, pages 201--218, 1996.

[43]

K. R. Scherer. Appraisal considered as a process of multi-level sequential checking. Appraisal processes in emotion: Theory, Methods, Research. Oxford University Press, New York and Oxford, k. r. scherer, a. schorr,&t. johnstone (eds.) edition, 2001.

[44]

K. R. Scherer. What are emotions? and how can they be measured? Social Science Information, 44(4):695--729, December 2005.

[45]

A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and trecvid. In MIR '06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pages 321--330, New York, NY, USA, 2007. ACM Press.

Digital Library

[46]

A. F. Smeaton and S. Rothwell. Biometric responses to music-rich segments in films: The cdvplex. In Seventh International Workshop on Content-Based Multimedia Indexing, pages 162--168, 2009.

Digital Library

[47]

M. Smeulders, A., M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. In IEEE Transactions on Pattern Analysis and Machine Intel ligence, volume 22, pages 1349--1380. IEEE Computer Society, 2000.

Digital Library

[48]

M. Soleymani, G. Chanel, J. J. Kierkels, and T. Pun. Affective ranking of movie scenes using physiological signals and content analysis. In MS '08: Proceeding of the 2nd ACM workshop on Multimedia semantics, pages 32--39, New York, NY, USA, 2008. ACM.

Digital Library

[49]

C. Tenopir, P. Wang, Y. Zhang, B. Simmons, and R. Pollard. Academic users' interactions with sciencedirect in search tasks: Affective and cognitive behaviors. Information Processing and Management: an International Journal, 44(1):105--121, 2008.

Digital Library

[50]

J. Urban and J. Jose, M. Evaluating a workspace's usefulness for image retrieval. Journal of Multimedia Systems, 12(4-5):355--373, 2007.

Digital Library

[51]

R. Valenti, N. Sebe, and T. Gevers. Facial expression recognition: A fully integrated approach. Image Analysis and Processing Workshops, 2007. ICIAPW 2007. 14th International Conference on, pages 125--130, Sept. 2007.

Digital Library

[52]

A. Vinciarelli, N. Suditu, and M. Pantic. Implicit human-centred tagging. In Proceedings of IEEE International Conference on Multimedia and Expo, 2009.

Digital Library

[53]

S. Vrochidis, P. King, L. Makris, A. Moumtzidou, V. Mezaris, and I. Kompatsiaris. Mklab interactive video retrieval system. In CIVR '08: Proceedings of the 2008 international conference on Content-based image and video retrieval, pages 563--564. ACM, 2008.

Digital Library

[54]

P. Wang, B. Hawk, W., and C. Tenopir. Users' interaction with world wide web resources: an exploratory study using a holistic approach. Information Processing and Management: an International Journal, 36(2):229--251, 2000.

Digital Library

[55]

G. M. Wilson and M. A. Sasse. Listen to your heart rate: counting the cost of media quality. pages 9--20, 2000.

Digital Library

Cited By

Ji KHettiachchi DSalim FScholer FSpina DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Characterizing Information Seeking Processes with Multiple Physiological SignalsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657793(1006-1017)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657793
Farooq SKim K(2024)Game Player ModelingEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_14(774-778)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_14
Ji KSpina DHettiachchi DScholer FSalim F(2023)Towards Detecting Tonic Information Processing Activities with Physiological DataAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610679(1-5)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610679
Show More Cited By

Index Terms

Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance

Recommendations

A comparison of general vs personalised affective models for the prediction of topical relevance
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Information retrieval systems face a number of challenges, originating mainly from the semantic gap problem. Implicit feedback techniques have been employed in the past to address many of these issues. Although this was a step towards the right ...
An effective implicit relevance feedback technique using affective, physiological and behavioural features
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

The effectiveness of various behavioural signals for implicit relevance feedback models has been exhaustively studied. Despite the advantages of such techniques for a real time information retrieval system, most of the behavioural signals are noisy and ...
Enriching user profiling with affective features for the improvement of a multimodal recommender system
CIVR '09: Proceedings of the ACM International Conference on Image and Video Retrieval

Recommender systems have been systematically applied in industry and academia to help users cope with information uncertainty. However, given the multiplicity of the preferences and needs it has been shown that no approach is suitable for all users in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '09: Proceedings of the 17th ACM international conference on Multimedia

October 2009

1202 pages

ISBN:9781605586083

DOI:10.1145/1631272

General Chairs:
Wen Gao
Peking University, China
,
Yong Rui
Microsoft, China
,
Alan Hanjalic
Delft University of Technology, The Netherlands
,
Program Chairs:
Changsheng Xu
Institute of Automation, Chinese Academy of Sciences, China
,
Eckehard Steinbach
Technical University of Munich, Germany
,
Abdulmotaleb El Saddik
University of Ottawa, Canada
,
Michelle Zhou
IBM T. J. Watson Research Center, USA

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM09

Sponsor:

SIGMM

MM09: ACM Multimedia Conference

October 19 - 24, 2009

Beijing, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

52
Total Citations
View Citations
765
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)4

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ji KHettiachchi DSalim FScholer FSpina DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Characterizing Information Seeking Processes with Multiple Physiological SignalsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657793(1006-1017)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657793
Farooq SKim K(2024)Game Player ModelingEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_14(774-778)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_14
Ji KSpina DHettiachchi DScholer FSalim F(2023)Towards Detecting Tonic Information Processing Activities with Physiological DataAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610679(1-5)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610679
Ji KSpina DHettiachchi DSalim FScholer FChen HDuh WHuang HKato MMothe JPoblete B(2023)Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing ActivitiesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591981(1971-1975)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591981
Polignano MNarducci FGemmis MSemeraro G(2020)Towards Emotion-aware Recommender Systems: an Affective Coherence Model based on Emotion-driven BehaviorsExpert Systems with Applications10.1016/j.eswa.2020.114382(114382)Online publication date: Dec-2020
https://doi.org/10.1016/j.eswa.2020.114382
Moshfeghi YTriantafillou PPollick F(2019)Towards Predicting a Realisation of an Information Need based on Brain SignalsThe World Wide Web Conference10.1145/3308558.3313671(1300-1309)Online publication date: 13-May-2019
https://dl.acm.org/doi/10.1145/3308558.3313671
Hassib MBraun MPfleging BAlt F(2019)Detecting and Influencing Driver Emotions Using Psycho-Physiological Sensors and Ambient LightHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29381-9_43(721-742)Online publication date: 2-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-29381-9_43
Bielozorov ABezbradica MHelfert M(2019)The Role of User Emotions for Content Personalization in e-Commerce: Literature ReviewHCI in Business, Government and Organizations. eCommerce and Consumer Behavior10.1007/978-3-030-22335-9_12(177-193)Online publication date: 14-Jun-2019
https://doi.org/10.1007/978-3-030-22335-9_12
Jasiewicz JKisilowska MJupowicz-Ginalska A(2019)Biometric Tools in Information Science. The Example of an Information Literacy Study – A Holiday Planning ExperimentInformation Literacy in Everyday Life10.1007/978-3-030-13472-3_3(23-32)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-030-13472-3_3
Jacucci GBarral ODaee PWenzel MSerim BRuotsalo TPluchino PFreeman JGamberini LKaski SBlankertz B(2019)Integrating neurophysiologic relevance feedback in intent modeling for information retrievalJournal of the Association for Information Science and Technology10.1002/asi.2416170:9(917-930)Online publication date: 2-Aug-2019
https://dl.acm.org/doi/10.1002/asi.24161
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents