More Web Proxy on the site http://driver.im/

article

Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Authors:

Luluah Al-Husain,

Alan ChamberlainAuthors Info & Claims

Personal and Ubiquitous Computing, Volume 19, Issue 7

Pages 1197 - 1212

https://doi.org/10.1007/s00779-015-0842-3

Published: 01 October 2015 Publication History

Abstract

Pervasive sensing has opened up new opportunities for measuring our feelings and understanding our behavior by monitoring our affective states while mobile. This review paper surveys pervasive affect sensing by examining and considering three major elements of affective pervasive systems, namely "sensing," "analysis," and "application." Sensing investigates the different sensing modalities that are used in existing real-time affective applications, analysis explores different approaches to emotion recognition and visualization based on different types of collected data, and application investigates different leading areas of affective applications. For each of the three aspects, the paper includes an extensive survey of the literature and finally outlines some of challenges and future research opportunities of affective sensing in the context of pervasive computing.

References

[1]

Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th international colloquium on signal processing and its applications (CSPA), pp 410---415

[2]

Lang PJ (1995) The emotion probe: studies of motivation and attention. Am Psychol 50:372---385

[3]

Mauss IB, Robinson MD (2009) Measures of emotion: a review. Cogn Emot 23:209---237

[4]

Peter C, Herbon A (2006) Emotion representation and physiology assignments in digital systems. Interact Comput 18:139---170

[5]

Picard RW, Papert S, Bender W, Blumberg B, Breazeal C, Cavallo D, Machover T, Resnick M, Roy D, Strohecker C (2004) Strohecker, affective learning--a manifesto. BT Technol J 22:253---269

[6]

Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25:49---59

[7]

Desmet P, Overbeeke K, Tax S (2001) Designing products with added emotional value: development and application of an approach for research through design. Des J 4:32---47

[8]

MacKerron G (2011) mappiness.org.uk

[9]

Mody RN, Willis KS, Kerstein V (2009) WiMo: location-based emotion tagging. In: Proceedings of the 8th international conference on mobile and ubiquitous multimedia, ACM, p 14

[10]

Affectiva Q. http://www.affectiva.com/q-sensor/

[11]

BioHarness. http://www.zephyranywhere.com/products/bioharness-3/

[12]

Bodymonitor Systeme (BMS). http://www.bodymonitor.de/

[13]

NeuroSky. http://www.neurosky.com/

[14]

Ayzenberg Y, Rivera JH, Picard R (2012) FEEL: frequent EDA and event logging--a mobile social interaction stress monitoring system. In: CHI `12 extended abstracts on human factors in computing systems, ACM, Austin, pp 2357---2362

[15]

Ståhl A, Höök K, Svensson M, Taylor AS, Combetto M (2009) Experiencing the affective diary. Personal Ubiquitous Comput 13:365---378

[16]

Yamamoto J, Kawazoe M, Nakazawa J, Takashio K, Tokuda H (2009) MOLMOD: analysis of feelings based on vital information for mood acquisition. Skin 4:6

[17]

Perttula A, Tuomi P, Suominen M, Koivisto A, Multisilta J (2010) Users as sensors: creating shared experiences in co-creational spaces by collective heart rate. In: Proceedings of the 14th international academic MindTrek conference: envisioning future media environments, ACM, pp 41---48

[18]

Matsumoto D, Willingham B (2009) Spontaneous facial expressions of emotion of congenitally and non-congenitally blind individuals. J Pers Soc Psychol 96(1):1---10

[19]

Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: Proceedings of the 2006 conference on computer vision and pattern recognition workshop, June 2006

[20]

Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Recognition and workshops, Santa Barbara, pp 866---871

[21]

Affdex. http://www.affectiva.com/affdex/

[22]

Hernandez J Facereader Noduls

[23]

Hoque ME, Drevo W, Picard RW (2012) Mood meter: counting smiles in the wild. In: Proceedings of the 2012 ACM conference on ubiquitous computing, ACM, pp 301---310

[24]

Calvo RA, D'Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. Affect Comput IEEE Trans 1:18---37

[25]

Juslin PN, Scherer KR (2005) Vocal expression of affect. In: Harrigan JA, Rosenthal R, Scherer KR (eds) The new handbook of methods in nonverbal behavior research. Oxford University Press, New York, pp 65---135

[26]

Bänziger T, Scherer KR (2005) The role of intonation in emotional expression. Speech Commun 46:252---267

[27]

Lee C, Narayanan S, Pieraccini R (2002) Classifying emotions in human-machine spoken dialogs. Presented at proceedings of international conference on multimedia and Expo, Lausanne, Switzerland, August 2002

[28]

Lu H, Rabbi M, Chittaranjan GT, Frauendorfer D, Mast MS, Campbell AT, Gatica-Perez D, Choudhury T (2012) StressSense: detecting stress in unconstrained acoustic environments using smartphones. In: Proceedings of 14th international conference ubiquitous computing

[29]

Rachuri KK, Musolesi M, Mascolo C, Rentfrow PJ, Longworth C, Aucinas A (2010) EmotionSense: a mobile phones based adaptive platform for experimental social psychology research. In: Proceedings of the 12th ACM international conference on ubiquitous computing, ACM, 2010, pp 281---290

[30]

LiKamWa R, Liu Y, Lane ND, Zhong L (2011) Can your smartphone infer your mood. In: PhoneSense workshop, 2011

[31]

Yuanchao M, Bin X, Yin B, Guodong S, Run Z (2012) Daily mood assessment based on mobile phone sensing. In: Wearable and implantable body sensor networks (BSN), 2012 ninth international conference on, 2012, pp 142---147

[32]

Mislove A, Lehmann S, Ahn Y, Onnela J, Rosenquist J (2010) Pulse of the nation: US mood throughout the day inferred from Twitter 2010

[33]

Noulas A, Mascolo C, Frias-Martinez E (2013) Exploiting foursquare and cellular data to infer user activity in urban environments

[34]

Kanjo E, El-Mawass N, Craveiro J, Ramos F (2013) Social, disconnected or in between: mobile data reveals urban mood. In: The 3rd international conference on the analysis of mobile phone datasets (NetMob'13), MIT, MA, p 9

[35]

André E (2011) Experimental methodology in emotion-oriented computing. Pervasive Comput IEEE 10:54---57

[36]

Lee H, Choi YS, Lee S, Park I (2012) Towards unobtrusive emotion recognition for affective social communication. In: Consumer communications and networking conference (CCNC), 2012 IEEE, IEEE, pp. 260---264

[37]

Oh K, Park H-S, Cho S-B (2010) A mobile context sharing system using activity and emotion recognition with Bayesian networks. In: 2010 7th international conference on ubiquitous intelligence & computing and 7th international conference on autonomic & trusted computing (UIC/ATC), IEEE, 2010, pp 244---249

[38]

Kim H-J, Choi YS (2011) EmoSens: affective entity scoring, a novel service recommendation framework for mobile platform. In: Workshop on personalization in mobile application of the 5th international conference on recommender system

[39]

Al-Barrak L, Kanjo E (2013) NeuroPlace: making sense of a place. In: Proceedings of the 4th augmented human international conference, ACM, pp 186---189

[40]

Chang DFK-h, Canny J (2011) Ammon: a speech analysis library for analyzing affect, stress, and mental health on mobile phones. In: Proceedings of PhoneSense, 2011

[41]

Chang K-h, Fisher D, Canny J (2011) Bj, 246, r. Hartmann, How's my mood and stress?: an efficient speech analysis library for unobtrusive monitoring on mobile phones. In: Proceedings of the 6th international conference on body area networks, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Beijing, 2011, pp 71---77

[42]

Patil SA, Hansen JH (2007) Speech under stress: analysis, modeling and recognition

[43]

Ekman P, Friesen WV Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

[44]

Ellsworth PC, Smith CA (1988) From appraisal to emotion: differences among unpleasant feelings. Motiv Emot 12(1978):271---302

[45]

Ekman P (1992) An argument for basic emotions. Cogn Emot 6:169---200

[46]

Küblbeck C, Ernst A (2006) Face detection and tracking in video sequences using the modifiedcensus transformation. Image Vis Comput 24:564---572

[47]

Wilhelmer R, Bismarck J, Maus B (2008) Feel-o-meter: Stimmungsgasometer. http://www.fühlometer.de

[48]

Bradley MM, Lang PJ (1999) Affective norms for English words (ANEW): instruction manual and affective ratings. In: Technical report C-1, the center for research in psychophysiology, University of Florida, 1999

[49]

Thelwall Mike, Buckley Kevan, Paltoglou Georgios, Cai Di, Kappas Arvid (2010) Sentiment in short strength detection informal text. JASIST 61(12):2544---2558

[50]

Matei S, Ball-Rokeach SJ, Qiu JL (2001) Fear and misperception of Los Angeles urban space: a spatial-statistical study of communication-shaped mental maps. Commun Res 28:429---463

[51]

Nold C (2006) Bio mapping. http://biomapping.net/

[52]

Pressman SD, Cohen S (2005) Does positive affect influence health? Psychol Bull 131:925

[53]

Fogg B (2007) Mobile persuasion: 20 perspectives on the future of behavior change. Mobile Persuasion 2007

[54]

Gay G, Pollak J, Adams P, Leonard J (2011) Pilot study of aurora, a social, mobile-phone-based emotion sharing and recording system. J Diabetes Sci Technol 5:325---332

[55]

Chang T-R, Kaasinen E, Kaipainen K (2013) Persuasive design in mobile applications for mental well-being: multidisciplinary expert review. In: Godara B, Nikita K (eds) Wireless mobile communication and healthcare. Springer, Berlin Heidelberg, pp 154---162

[56]

Dickerson RF, Gorlin EI, Stankovic JA (2011) Empath: a continuous remote emotional health monitoring system for depressive illness. In: Proceedings of the 2nd conference on wireless health, ACM, 2011, p 5

[57]

O.A.P. Ltd., Optimism apps (2011)

[58]

Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, Mohr DC (2011) Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res 13(3):e55

[59]

Gaggioli A, Pioggia G, Tartarisco G, Baldus G, Corda D, Cipresso P, Riva G (2013) A mobile data collection platform for mental health research. Personal Ubiquitous Comput 17:241---251

[60]

Morris ME, Kathawala Q, Leen TK, Gorenstein EE, Guilak F, Labhard M, Deleeuw W (2010) Mobile therapy: case study evaluations of a cell phone application for emotional self-awareness. J Med Internet Res 12(2):e10

[61]

Bergner BS, Exner J-P, Zeile P, Rumberg M (2012) Sensing the city---how to identify recreational benefits of urban green areas with the help of sensor technology. In: Proceedings REAL CORP, 2012

[62]

Klettner S, Huang H, Schmidt M, Gartner G (2013) Crowdsourcing affective responses to space. KN Kartographische Nachrichten. J Cartogr Geogr Inf 2

[63]

Gartner G (2012) Openemotionmap. org--Emotional response to space as an additional concept in cartography. Int Arch Photogramm Remote Sens Spat Inf Sci (ISPRS) 39-B4:473---476

[64]

Doerflinger J, Gross T, Lyra O, Karapanos E, Kostakos V, Cuadrado-Cordero I, Soria-Morillo LM, Gonzalez-Abril L, Ortega-Ramirez JA, Pau de la Cruz I (2012) ICTD work, plus mFeel, pervasive computing, IEEE, vol 11, pp 43---45. Replace with Sense of space paper

[65]

Killingsworth MA, Gilbert DT (2010) A wandering mind is an unhappy mind. Science 330:932

[66]

Christin D, Reinhardt A, Kanhere SS, Hollick M (2011) A survey on privacy in mobile participatory sensing applications. J Syst Softw 84:1928---1946

[67]

Crabtree A, Chamberlain A, Grinter R, Jones M, Rodden T, Rogers Y (eds) (2013) Special issue on "The Turn to the Wild Introduction". ACM Transactions on Computer-Human Interaction--ToCHI 20(3), 13:1---13:4

[68]

Kanjo E, Alajmi N, El-Mawass N (2013) ShopMobia: emotion based shop rating system, affective computing and intelligent interaction conference. In Submission, Geneva, Switzerland

[69]

Lane ND, Xu Y, Lu H, Hu S, Choudhury T, Campbell AT, Zhao F (2011) Enabling large-scale human activity inference on smartphones using community similarity networks (csn). In: Proceedings of the 13th international conference on Ubiquitous computing, ACM, 2011, pp 355---364

[70]

Li Q, Cao G (2013) Providing privacy-aware incentives for mobile sensing. In: IEEE international conference on pervasive computing and communications (PerCom), 2013

[71]

Ra M-R, Priyantha B, Kansal A, Liu J (2012) Improving energy efficiency of personal sensing applications with heterogeneous multi-processors. In: Proceedings of the 2012 ACM conference on ubiquitous computing (UbiComp `12). ACM, New York, pp 1---10.

[72]

Chamberlain A, Crabtree A, Rodden T, Jones M, Rogers Y (201) Research in the wild: understanding `in the wild' approaches to design and development. In: Conference on designing interactive systems 2012 ACM DIS, ACM Press

[73]

Crabtree A, Chamberlain A, Grinter RR, Jones M, Rodden T, Rogers Y (eds) (2013) Special issue on "The Turn to the Wild" with authored introduction". ACM Transactions on Computer-Human Interaction--ToCHI 20(3), 13:1---13:4

[74]

Crabtree A, Chamberlain A, Davies M, Glover K, Reeves S, Rodden T, Tolmie P, Jones M (2013) "Doing Innovation in the Wild", CHItaly 2013. Trento, Italy--(ACM Library)

[75]

Tolmie P, Chamberlain A, Benford S (2014) Designing for Reportability: sustainable gamification, public engagement and promoting environmental debate. In Personal and ubiquitous computing journal, Springer, New York

Cited By

Ren BTsiu DNitsche M(2024)Enhancing Social Engagement in Cinemas through the Promotion of Emotion-SharingProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3663638(361-368)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3639701.3663638
Yang HLee C(2023)A Media-Guided Attentive Graphical Network for Personality Recognition Using PhysiologyIEEE Transactions on Affective Computing10.1109/TAFFC.2021.309004014:2(931-943)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3090040
Abu Shaqra FDuwairi RAl-Ayyoub M(2022)A multi-modal deep learning system for Arabic emotion recognitionInternational Journal of Speech Technology10.1007/s10772-022-09981-w26:1(123-139)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1007/s10772-022-09981-w
Show More Cited By

Index Terms

Emotions in context: examining pervasive affective sensing systems, applications, and analyses

Recommendations

Emotions on the Go: Mobile Emotion Assessment in Real-Time using Facial Expressions
AVI '20: Proceedings of the 2020 International Conference on Advanced Visual Interfaces

Exploiting emotions for user interface evaluation became an increasingly important research objective in Human-Computer Interaction. Emotions are usually assessed through surveys that do not allow information to be collected in real-time. In our work, we ...
AVEC 2014: 3D Dimensional Affect and Depression Recognition Challenge
AVEC '14: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence, arousal and dominance. In ...
AVEC 2013: the continuous audio/visual emotion and depression recognition challenge
AVEC '13: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge

Mood disorders are inherently related to emotion. In particular, the behaviour of people suffering from mood disorders such as unipolar depression shows a strong temporal correlation with the affective dimensions valence and arousal. In addition, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Personal and Ubiquitous Computing

Personal and Ubiquitous Computing Volume 19, Issue 7

October 2015

235 pages

ISSN:1617-4909

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer-Verlag London.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 October 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
82
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ren BTsiu DNitsche M(2024)Enhancing Social Engagement in Cinemas through the Promotion of Emotion-SharingProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3663638(361-368)Online publication date: 7-Jun-2024
https://dl.acm.org/doi/10.1145/3639701.3663638
Yang HLee C(2023)A Media-Guided Attentive Graphical Network for Personality Recognition Using PhysiologyIEEE Transactions on Affective Computing10.1109/TAFFC.2021.309004014:2(931-943)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TAFFC.2021.3090040
Abu Shaqra FDuwairi RAl-Ayyoub M(2022)A multi-modal deep learning system for Arabic emotion recognitionInternational Journal of Speech Technology10.1007/s10772-022-09981-w26:1(123-139)Online publication date: 21-Jun-2022
https://dl.acm.org/doi/10.1007/s10772-022-09981-w
Hoefer MVan Kleunen LGoodby CBlackburn LPanati PVoida S(2021)The Multiplicative Patient and the Clinical Workflow: Clinician Perspectives on Social Interfaces for Self-Tracking and Managing Bipolar DisorderProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3461995(907-925)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3461995
Schroeder JSuh JWilks CCzerwinski MMunson SFogarty JAlthoff TArriaga R(2020)Data-Driven Implications for Translating Evidence-Based Psychotherapies into Technology-Delivered InterventionsProceedings of the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare10.1145/3421937.3421975(274-287)Online publication date: 18-May-2020
https://dl.acm.org/doi/10.1145/3421937.3421975
Zhang TEl Ali AWang CHanjalic ACesar PBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)RCEA: Real-time, Continuous Emotion Annotation for Collecting Precise Mobile Video Ground Truth LabelsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376808(1-15)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376808
Wampfler RKlingler SSolenthaler BSchinazi VGross MBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Affective State Prediction Based on Semi-Supervised Learning from Smartphone Touch DataProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376504(1-13)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376504
Younis EKanjo EChamberlain A(2019)Designing and evaluating mobile self-reporting techniquesPersonal and Ubiquitous Computing10.1007/s00779-019-01207-223:2(329-338)Online publication date: 1-Apr-2019
https://dl.acm.org/doi/10.1007/s00779-019-01207-2
Cohen YConstantinides MMarshall P(2019)Places for News: A Situated Study of Context in News ConsumptionHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29384-0_5(69-91)Online publication date: 2-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-29384-0_5
Nave CRomão TCorreia N(2018)Self-tracking emotional states through social media mobile photographyProceedings of the 32nd International BCS Human Computer Interaction Conference10.14236/ewic/HCI2018.146(1-6)Online publication date: 4-Jul-2018
https://dl.acm.org/doi/10.14236/ewic/HCI2018.146
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents