[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3338286.3340143acmconferencesArticle/Chapter ViewAbstractPublication PagesmobilehciConference Proceedingsconference-collections
research-article

Hand-Over-Face Input Sensing for Interaction with Smartphones through the Built-in Camera

Published: 01 October 2019 Publication History

Abstract

This paper proposes using face as a touch surface and employing hand-over-face (HOF) gestures as a novel input modality for interaction with smartphones, especially when touch input is limited. We contribute InterFace, a general system framework that enables the HOF input modality using advanced computer vision techniques. As an examplar of the usage of this framework, we demonstrate the feasibility and usefulness of HOF with an Android application for improving single-user and group selfie-taking experience through providing appearance customization in real-time. In a within-subjects study comparing HOF against touch input for single-user interaction, we found that HOF input led to significant improvements in accuracy and perceived workload, and was preferred by the participants. Qualitative results of an observational study also demonstrated the potential of HOF input modality to improve the user experience in multi-user interactions. Based on the lessons learned from our studies, we propose a set of potential applications of HOF to support smartphone interaction. We envision that the affordances provided by the this modality can expand the mobile interaction vocabulary and facilitate scenarios where touch input is limited or even not possible.

References

[1]
[n. d.]. BMW's Gesture Control. https://driving.ca/bmw/7-series/auto-news/news/how-it-works-bmw-gesture-control. Accessed: 2018-09-18.
[2]
[n. d.]. Mobile Vision. https://developers.google.com/vision/. Accessed: 2018-09-17.
[3]
Gilles Bailly, Jörg Müller, Michael Rohs, Daniel Wigdor, and Sven Kratz. 2012. ShoeSense: A New Perspective on Gestural Interaction and Wearable Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1239--1248. https://doi.org/10.1145/2207676.2208576
[4]
Cynthia L. Bennett, Jane E, Martez E. Mott, Edward Cutrell, and Meredith Ringel Morris. 2018. How Teens with Visual Impairments Take, Edit, and Share Photos on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 76, 12 pages. https://doi.org/10.1145/3173574.3173650
[5]
Lung-Pan Cheng, Fang-I Hsiao, Yen-Ting Liu, and Mike Y. Chen. 2012. iRotate: Automatic Screen Rotation Based on Face Orientation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2203--2210. https://doi.org/10.1145/2207676.2208374
[6]
Shaowei Chu and Jiro Tanaka. 2011. Hand Gesture for Taking Self Portrait. In Human-Computer Interaction. Interaction Techniques and Environments, Julie A. Jacko (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 238--247.
[7]
S. Chu, F. Zhang, N.Ji, Z.Jin, and R. Pan. 2017. Pan-and-tilt self-portrait system using gesture interface. In 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS). 599--605. https://doi.org/10.1109/ICIS.2017.7960063
[8]
Alexander De Luca, Alina Hang, Emanuel von Zezschwitz, and Heinrich Hussmann. 2015. I Feel Like I'm Taking Selfies All Day!: Towards Understanding Biometric Authentication on Smartphones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1411--1414. https://doi.org/10.1145/2702123.2702141
[9]
Eva Eriksson, Thomas Riisgaard Hansen, and Andreas Lykke-Olesen. 2007. Movement-based Interaction in Camera Spaces: A Conceptual Framework. Personal Ubiquitous Comput. 11, 8 (Dec. 2007), 621--632. https://doi.org/10.1007/s00779-006-0134-z
[10]
Jun Gong, Zheer Xu, Qifan Guo, Teddy Seyed, Xiang 'Anthony' Chen, Xiaojun Bi, and Xing-Dong Yang. 2018. WrisText: One-handed Text Entry on Smartwatch Using Wrist Gestures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 181, 14 pages. https://doi.org/10.1145/3173574.3173755
[11]
Sukeshini A. Grandhi, Gina Joue, and Irene Mittelberg. 2011. Understanding Naturalness and Intuitiveness in Gesture Production: Insights for Touchless Gestural Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 821--824. https://doi.org/10.1145/1978942.1979061
[12]
Sean G. Gustafson, Bernhard Rabe, and Patrick M. Baudisch. 2013. Understanding Palm-based Imaginary Interfaces: The Role of Visual and Tactile Cues when Browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 889--898. https://doi.org/10.1145/2470654.2466114
[13]
Thomas Riisgaard Hansen, Eva Eriksson, and Andreas Lykke-Olesen. 2006. Use Your Head: Exploring Face Tracking for Mobile Interaction. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 845--850. https://doi.org/10.1145/1125451.1125617
[14]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139--183. https://doi.org/10.1016/S0166-4115(08)62386-9
[15]
Ken Hinckley and Hyunyoung Song. 2011. Sensor Synaesthesia: Touch in Motion, and Motion in Touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 801--810. https://doi.org/10.1145/1978942.1979059
[16]
Ken Hinckley and Daniel Wigdor. 2002. Input technologies and techniques. In The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. Taylor and Francis, Chapter 9.
[17]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
[18]
Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1526--1537. https://doi.org/10.1145/2858036.2858483
[19]
Y. Huang, X. Liu, L. Jin, and X. Zhang. 2015. DeepFinger: A Cascade Convolutional Neuron Network Approach to Finger Key Point Detection in Egocentric Vision with Mobile Camera. In 2015 IEEE International Conference on Systems, Man, and Cybernetics. 2944--2949. https://doi.org/10.1109/SMC.2015.512
[20]
Neel Joshi, Abhishek Kar, and Michael Cohen. 2012. Looking at You: Fused Gyro and Face Tracking for Viewing Large Imagery on Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2211--2220. https://doi.org/10.1145/2207676.2208375
[21]
S. K. Kang, M. Y. Nam, and P. K. Rhee. 2008. Color Based Hand and Finger Detection Technology for User Interaction. In 2008 International Conference on Convergence and Hybrid Information Technology. 229--236. https://doi.org/10.1109/ICHIT.2008.292
[22]
Hsin-Liu (Cindy) Kao, Artem Dementyev, Joseph A. Paradiso, and Chris Schmandt. 2015. NailO: Fingernails As an Input Surface. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3015--3018. https://doi.org/10.1145/2702123.2702572
[23]
James E Katz and Elizabeth Thomas Crocker. 2015. Selfies| selfies and photo messaging as visual conversation: Reports from the United States, United Kingdom and China. International Journal of Communication 9 (2015), 12.
[24]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere Using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 167--176. https://doi.org/10.1145/2380116.2380139
[25]
Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Luka Cehovin, Gustavo Fernandez, Tomas Vojir, Gustav Hager, Georg Nebehay, and Roman Pflugfelder. 2015. The Visual Object Tracking VOT2015 Challenge Results. In The IEEE International Conference on Computer Vision (ICCV) Workshops.
[26]
Manu Kumar and Terry Winograd. 2007. Gaze-enhanced Scrolling Techniques. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 213--216. https://doi.org/10.1145/1294211.1294249
[27]
Gierad Laput, Robert Xiao, Xiang 'Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-powered and Clickable Fixed-icon Laser Projectors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 389--394. https://doi.org/10.1145/2642918.2647356
[28]
Oliver Lemon. 2012. Conversational Interfaces. Springer New York, New York, NY, 1--4. https://doi.org/10.1007/978-1-4614-4803-7_1
[29]
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single Shot MultiBox Detector. In Computer Vision -- ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 21--37.
[30]
Alan Lukezic, Tomas Vojir, Luka Cehovin Zajc, Jiri Matas, and Matej Kristan. 2017. Discriminative Correlation Filter With Channel and Spatial Reliability. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[31]
Marwa Mahmoud, Tadas Baltrušaitis, Peter Robinson, and Laurel D. Riek. 2011. 3D Corpus of Spontaneous Complex Mental States. In Affective Computing and Intelligent Interaction, Sidney D'Mello, Arthur Graesser, Björn Schuller, and Jean-Claude Martin (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 205--214.
[32]
Marwa Mahmoud and Peter Robinson. 2011. Interpreting Hand-Over-Face Gestures. In Affective Computing and Intelligent Interaction. Springer Berlin Heidelberg, Berlin, Heidelberg, 248--255.
[33]
Emiliano Miluzzo, Tianyu Wang, and Andrew T. Campbell. 2010. EyePhone: Activating Mobile Phones with Your Eyes. In Proceedings of the Second ACM SIGCOMM Workshop on Networking, Systems, and Applications on Mobile Handhelds (MobiHeld '10). ACM, New York, NY, USA, 15--20. https://doi.org/10.1145/1851322.1851328
[34]
Pranav Mistry and Pattie Maes. 2009. SixthSense: A Wearable Gestural Interface. In ACM SIGGRAPH ASIA 2009 Sketches (SIGGRAPH ASIA '09). ACM, New York, NY, USA, Article 11, 1 pages. https://doi.org/10.1145/1667146.1667160
[35]
Behnaz Nojavanasghari, Charles E. Hughes, Tadas Baltrusaitis, and Louis-Philippe Morency. 2017. Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions. CoRR abs/1708.00370 (2017). arXiv:1708.00370 http://arxiv.org/abs/1708.00370
[36]
Jagdish Lal Raheja, Karen Das, and Ankit Chaudhary. 2012. Fingertip Detection: A Fast Method with Natural Hand. CoRR abs/1212.0134 (2012). arXiv:1212.0134 http://arxiv.org/abs/1212.0134
[37]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[38]
I. Scott MacKenzie and Behrooz Ashtiani. 2011. BlinkWrite: efficient text entry using eye blinks. Universal Access in the Information Society 10, 1 (01 Mar 2011), 69--80. https://doi.org/10.1007/s10209-010-0188-6
[39]
Marcos Serrano, Barrett M. Ens, and Pourang P. Irani. 2014. Exploring the Use of Hand-to-face Input for Interacting with Head-worn Displays. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3181--3190. https://doi.org/10.1145/2556288.2556984
[40]
Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On- and Above-Skin Input Sensing Through a Wearable Depth Sensor. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3891--3902. https://doi.org/10.1145/3025453.3026005
[41]
Robert Walter, Gilles Bailly, Nina Valkanova, and Jörg Müller. 2014. Cuenesics: Using Mid-air Gestures to Select Items on Interactive Public Displays. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '14). ACM, New York, NY, USA, 299--308. https://doi.org/10.1145/2628363.2628368
[42]
Cheng-Yao Wang, Min-Chieh Hsiu, Po-Tsung Chiu, Chiao-Hui Chang, Liwei Chan, Bing-Yu Chen, and Mike Y. Chen. 2015. PalmGesture: Using Palms As Gesture Interfaces for Eyes-free Input. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 217--226. https://doi.org/10.1145/2785830.2785885
[43]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 143--146. https://doi.org/10.1145/1978942.1978963
[44]
Koki Yamashita, Takashi Kikuchi, Katsutoshi Masai, Maki Sugimoto, Bruce H. Thomas, and Yuta Sugiura. 2017. CheekInput: Turning Your Cheek into an Input Surface by Embedded Optical Sensors on a Head-mounted Display. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, New York, NY, USA, Article 19, 8 pages. https://doi.org/10.1145/3139131.3139146
[45]
Jian Zhao, Ricardo Jota, Daniel J. Wigdor, and Ravin Balakrishnan. 2016. Augmenting Mobile Phone Interaction with Face-Engaged Gestures. CoRR abs/1610.00214 (2016). arXiv:1610.00214 http://arxiv.org/abs/1610.00214

Cited By

View all
  • (2024)iFace: Hand-Over-Face Gesture Recognition Leveraging Impedance SensingProceedings of the Augmented Humans International Conference 202410.1145/3652920.3652923(131-137)Online publication date: 4-Apr-2024
  • (2023)D-Touch: Recognizing and Predicting Fine-grained Hand-face Touching Activities Using a Neck-mounted WearableProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584063(569-583)Online publication date: 27-Mar-2023
  • (2023)Masktrap: Designing and Identifying Gestures to Transform Mask Strap into an Input InterfaceProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584062(762-775)Online publication date: 27-Mar-2023
  • Show More Cited By

Index Terms

  1. Hand-Over-Face Input Sensing for Interaction with Smartphones through the Built-in Camera

        Recommendations

        Comments

        Please enable JavaScript to view thecomments powered by Disqus.

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        MobileHCI '19: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services
        October 2019
        646 pages
        ISBN:9781450368254
        DOI:10.1145/3338286
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 01 October 2019

        Permissions

        Request permissions for this article.

        Check for updates

        Badges

        • Best Paper

        Author Tags

        1. Computer vision
        2. input modality
        3. smartphones

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        MobileHCI '19
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 202 of 906 submissions, 22%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)72
        • Downloads (Last 6 weeks)3
        Reflects downloads up to 05 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)iFace: Hand-Over-Face Gesture Recognition Leveraging Impedance SensingProceedings of the Augmented Humans International Conference 202410.1145/3652920.3652923(131-137)Online publication date: 4-Apr-2024
        • (2023)D-Touch: Recognizing and Predicting Fine-grained Hand-face Touching Activities Using a Neck-mounted WearableProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584063(569-583)Online publication date: 27-Mar-2023
        • (2023)Masktrap: Designing and Identifying Gestures to Transform Mask Strap into an Input InterfaceProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584062(762-775)Online publication date: 27-Mar-2023
        • (2023)Understanding Wheelchair Users’ Preferences for On-Body, In-Air, and On-Wheelchair GesturesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580929(1-16)Online publication date: 19-Apr-2023
        • (2023)OmniSense: Exploring Novel Input Sensing and Interaction Techniques on Mobile Device with an Omni-Directional CameraProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580747(1-18)Online publication date: 19-Apr-2023
        • (2022)Cross-Subject Activity Detection for COVID-19 Infection Avoidance Based on Automatically Annotated IMU DataIEEE Sensors Journal10.1109/JSEN.2022.317629122:13(13125-13135)Online publication date: 1-Jul-2022
        • (2021)HandyTrak: Recognizing the Holding Hand on a Commodity Smartphone from Body Silhouette ImagesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474817(1210-1220)Online publication date: 10-Oct-2021
        • (2021)Exploring Pseudo Hand-Eye Interaction on the Head-Mounted DisplayProceedings of the Augmented Humans International Conference 202110.1145/3458709.3458960(251-258)Online publication date: 22-Feb-2021
        • (2021)FaceSight: Enabling Hand-to-Face Gesture Interaction on AR Glasses with a Downward-Facing Camera VisionProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445484(1-14)Online publication date: 6-May-2021
        • (2020)Face Commands - User-Defined Facial Gestures for Smart Glasses2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR50242.2020.00064(374-386)Online publication date: Nov-2020

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media