[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Personalized Facial Gesture Recognition for Accessible Mobile Gaming

  • Conference paper
  • First Online:
Computers Helping People with Special Needs (ICCHP 2024)

Abstract

For people with upper extremity motor impairments, interaction with mobile devices is challenging because it relies on the use of the touchscreen. Existing assistive solutions replace inaccessible touchscreen interactions with sequences of simpler and accessible ones. However, the resulting sequence takes longer to perform than the original interaction, and therefore it is unsuitable for mobile video games. In this paper, we expand our prior work on accessible interaction substitutions for video games with a new interaction modality: using facial gestures. Our approach allows users to play existing mobile video games using custom facial gestures. The gestures are defined by each user according to their own needs, and the system is trained with a small number of face gesture samples collected from the user. The recorded gestures are then mapped to the touchscreen interactions required to play a target game. Each interaction corresponds to a single face gesture, making this approach suitable for the interaction with video games. We describe the facial gesture recognition pipeline, motivating the implementation choices through preliminary experiments conducted on example videos of face gestures collected by one user without impairments. Preliminary results show that an accurate classification of facial gestures (\(97\%\)) is possible even with as few as 5 samples of the user.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 54.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 69.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    FGs considered in the experiment: smile, open mouth, close left/right eye, curve eybrows, wrinkle nose, turn left/right, incline left/right, raise/lower head.

  2. 2.

    Subset of Mediapipe Face Mesh [7] landmarks that are also defined in Dlib [8].

References

  1. Android accessibility services. https://developer.android.com/guide/topics/ui/accessibility/service

  2. Android accessibility suite. https://play.google.com/store/apps/details?id=com.google.android.marvin.talkback

  3. Ahmetovic, D., Riboli, D., Bernareggi, C., Mascetti, S.: RePlay: Touchscreen interaction substitution method for accessible gaming. In: Human Computer Interaction with Mobile Devices and Services. ACM (2021)

    Google Scholar 

  4. Bierre, K., Chetwynd, J., Ellis, B., Hinn, D.M., Ludi, S., Westin, T.: Game not over: Accessibility issues in video games. In: Universal Access in Human-Computer Interaction (2005)

    Google Scholar 

  5. Bishop, D.V., Brown, B.B., Robson, J.: The relationship between phoneme discrimination, speech production, and language comprehension in cerebral-palsied individuals. J. Speech Lang. Hear. Res. 33(2), 210–9 (1990)

    Google Scholar 

  6. Gu, Q., Li, Z., Han, J.: Generalized fisher score for feature selection. In: Conference on Uncertainty in Artificial Intelligence (2011)

    Google Scholar 

  7. Kartynnik, Y., Ablavatski, A., Grishchenko, I., Grundmann, M.: Real-time facial surface geometry from monocular video on mobile GPUs (2019). arXiv preprint arXiv:1907.06724

  8. King, D.E.: Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  9. Lecours, A., Lhermitte, F.: The “pure form” of the phonetic disintegration syndrome (pure anarthria); anatomo-clinical report of a historical case. Brain Lang. 3(1), 88–113 (1976)

    Google Scholar 

  10. Naftali, M., Findlater, L.: Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments. In: Computers & Accessibility. ACM (2014)

    Google Scholar 

  11. Revina, I.M., Emmanuel, W.S.: A survey on human face expression recognition techniques. J. King Saud Univ.-Comput. Inf. Sci. 33(6), 619–628 (2021)

    Google Scholar 

  12. Rozado, D., Niu, J., Lochner, M.: Fast human-computer interaction by combining gaze pointing and face gestures. Trans. Accessible Comput. 10(3), 1–18 (2017)

    Google Scholar 

  13. Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  14. Zhang, X., Kulkarni, H., Morris, M.R.: Smartphone-based gaze gesture communication for people with motor disabilities. In: Human Factors in Computing Systems. ACM (2017)

    Google Scholar 

  15. Zhao, X., Fan, M., Han, T.: “i don’t want people to look at me differently” designing user-defined above-the-neck gestures for people with upper body motor impairments. In: Human Factors in Computing Systems (2022)

    Google Scholar 

  16. Zhong, Y., Raman, T., Burkhardt, C., Biadsy, F., Bigham, J.P.: JustSpeak: enabling universal voice control on android. In: Web for All. ACM (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dragan Ahmetovic .

Editor information

Editors and Affiliations

Ethics declarations

Disclosure of Interests

The authors have no competing interests.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Manzoni, M., Ahmetovic, D., Mascetti, S. (2024). Personalized Facial Gesture Recognition for Accessible Mobile Gaming. In: Miesenberger, K., Peňáz, P., Kobayashi, M. (eds) Computers Helping People with Special Needs. ICCHP 2024. Lecture Notes in Computer Science, vol 14750. Springer, Cham. https://doi.org/10.1007/978-3-031-62846-7_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-62846-7_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-62845-0

  • Online ISBN: 978-3-031-62846-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics