Abstract
For people with upper extremity motor impairments, interaction with mobile devices is challenging because it relies on the use of the touchscreen. Existing assistive solutions replace inaccessible touchscreen interactions with sequences of simpler and accessible ones. However, the resulting sequence takes longer to perform than the original interaction, and therefore it is unsuitable for mobile video games. In this paper, we expand our prior work on accessible interaction substitutions for video games with a new interaction modality: using facial gestures. Our approach allows users to play existing mobile video games using custom facial gestures. The gestures are defined by each user according to their own needs, and the system is trained with a small number of face gesture samples collected from the user. The recorded gestures are then mapped to the touchscreen interactions required to play a target game. Each interaction corresponds to a single face gesture, making this approach suitable for the interaction with video games. We describe the facial gesture recognition pipeline, motivating the implementation choices through preliminary experiments conducted on example videos of face gestures collected by one user without impairments. Preliminary results show that an accurate classification of facial gestures (\(97\%\)) is possible even with as few as 5 samples of the user.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Android accessibility services. https://developer.android.com/guide/topics/ui/accessibility/service
Android accessibility suite. https://play.google.com/store/apps/details?id=com.google.android.marvin.talkback
Ahmetovic, D., Riboli, D., Bernareggi, C., Mascetti, S.: RePlay: Touchscreen interaction substitution method for accessible gaming. In: Human Computer Interaction with Mobile Devices and Services. ACM (2021)
Bierre, K., Chetwynd, J., Ellis, B., Hinn, D.M., Ludi, S., Westin, T.: Game not over: Accessibility issues in video games. In: Universal Access in Human-Computer Interaction (2005)
Bishop, D.V., Brown, B.B., Robson, J.: The relationship between phoneme discrimination, speech production, and language comprehension in cerebral-palsied individuals. J. Speech Lang. Hear. Res. 33(2), 210–9 (1990)
Gu, Q., Li, Z., Han, J.: Generalized fisher score for feature selection. In: Conference on Uncertainty in Artificial Intelligence (2011)
Kartynnik, Y., Ablavatski, A., Grishchenko, I., Grundmann, M.: Real-time facial surface geometry from monocular video on mobile GPUs (2019). arXiv preprint arXiv:1907.06724
King, D.E.: Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)
Lecours, A., Lhermitte, F.: The “pure form” of the phonetic disintegration syndrome (pure anarthria); anatomo-clinical report of a historical case. Brain Lang. 3(1), 88–113 (1976)
Naftali, M., Findlater, L.: Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments. In: Computers & Accessibility. ACM (2014)
Revina, I.M., Emmanuel, W.S.: A survey on human face expression recognition techniques. J. King Saud Univ.-Comput. Inf. Sci. 33(6), 619–628 (2021)
Rozado, D., Niu, J., Lochner, M.: Fast human-computer interaction by combining gaze pointing and face gestures. Trans. Accessible Comput. 10(3), 1–18 (2017)
Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Zhang, X., Kulkarni, H., Morris, M.R.: Smartphone-based gaze gesture communication for people with motor disabilities. In: Human Factors in Computing Systems. ACM (2017)
Zhao, X., Fan, M., Han, T.: “i don’t want people to look at me differently” designing user-defined above-the-neck gestures for people with upper body motor impairments. In: Human Factors in Computing Systems (2022)
Zhong, Y., Raman, T., Burkhardt, C., Biadsy, F., Bigham, J.P.: JustSpeak: enabling universal voice control on android. In: Web for All. ACM (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Manzoni, M., Ahmetovic, D., Mascetti, S. (2024). Personalized Facial Gesture Recognition for Accessible Mobile Gaming. In: Miesenberger, K., Peňáz, P., Kobayashi, M. (eds) Computers Helping People with Special Needs. ICCHP 2024. Lecture Notes in Computer Science, vol 14750. Springer, Cham. https://doi.org/10.1007/978-3-031-62846-7_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-62846-7_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-62845-0
Online ISBN: 978-3-031-62846-7
eBook Packages: Computer ScienceComputer Science (R0)