More Web Proxy on the site http://driver.im/

research-article

Phrase-Gesture Typing on Smartphones

Authors:

Xing-Dong YangAuthors Info & Claims

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

Article No.: 55, Pages 1 - 11

https://doi.org/10.1145/3526113.3545683

Published: 28 October 2022 Publication History

Abstract

We study phrase-gesture typing, a gesture typing method that allows users to type short phrases by swiping through all the letters of the words in a phrase using a single, continuous gesture. Unlike word-gesture typing, where text needs to be entered word by word, phrase-gesture typing enters text phrase by phrase. To demonstrate the usability of phrase-gesture typing, we implemented a prototype called PhraseSwipe. Our system is composed of a frontend interface designed specifically for typing through phrases and a backend phrase-level gesture decoder developed based on a transformer-based neural language model. Our decoder was trained using five million phrases of varying lengths of up to five words, chosen randomly from the Yelp Review Dataset. Through a user study with 12 participants, we demonstrate that participants could type using PhraseSwipe at an average speed of 34.5 WPM with a Word Error Rate of 1.1%.

References

[1]

American National Corpus. Retrieved April 2, 2022 from http://www.anc.org/.

[2]

Cornell Movie–Dialogs Corpus. Retrieved April 2, 2022 from https://www.cs.cornell.edu/∼cristian/Cornell_Movie-Dialogs_Corpus.html.

[3]

Gboard - the Google Keyboard. Retrieved April 2, 2022 from https://play.google.com/store/apps/details?id=com.google.android.inputmethod.latin&hl=en_US&gl=US.

[4]

Microsoft SwiftKey Keyboard. Retrieved April 2, 2022 from https://play.google.com/store/apps/details?id=com.touchtype.swiftkey&hl=en_US&gl=US.

[5]

Ouais Alsharif, Tom Ouyang, Françoise Beaufays, Shumin Zhai, Thomas Breuel and Johan Schalkwyk. 2015. Long short term memory neural network for keyboard gesture decoding. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2076-2080.

[6]

Xiaojun Bi, Shiri Azenkot, Kurt Partridge and Shumin Zhai. 2013. Octopus: evaluating touchscreen keyboard correction and recognition algorithms via remulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13), ACM, 543-552.

Digital Library

[7]

Xiaojun Bi, Ciprian Chelba, Tom Ouyang, Kurt Partridge and Shumin Zhai. 2012. Bimanual gesture keyboard. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST’12), ACM, 137-146.

Digital Library

[8]

Xiaojun Bi and Shumin Zhai. 2016. IJQwerty: What difference does one key change make? Gesture typing keyboard optimization bounded by one key position change from Qwerty. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16), ACM, 49-58.

Digital Library

[9]

Xiaojun Bi and Shumin Zhai. 2016. Predicting finger-touch accuracy based on the dual Gaussian distribution model. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST’16), 313-319.

Digital Library

[10]

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.

[11]

Ciprian Chelba, Mohammad Norouzi and Samy Bengio. 2017. N-gram language modeling using recurrent neural network estimation. arXiv preprint arXiv:1703.10724.

[12]

Sibo Chen, Junce Wang, Santiago Guerra, Neha Mittal and Soravis Prakkamakul. 2019. Exploring word-gesture text entry techniques in virtual reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA’19), Association for Computing Machinery, Glasgow, Scotland Uk, Paper LBW0233.

Digital Library

[13]

Wenzhe Cui, Jingjie Zheng, Blaine Lewis, Daniel Vogel and Xiaojun Bi. 2019. HotStrokes: word-gesture shortcuts on a trackpad. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19), Association for Computing Machinery, Paper 165.

Digital Library

[14]

Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

[15]

Joshua Goodman, Gina Venolia, Keith Steury and Chauncey Parker. 2002. Language modeling for soft keyboards. In Proceedings of the 7th international conference on Intelligent user interfaces (IUI’02), ACM, 194-195.

Digital Library

[16]

Asela Gunawardana, Tim Paek and Christopher Meek. 2010. Usability guided key-target resizing for soft keyboards. In Proceedings of the 15th international conference on Intelligent user interfaces (IUI’10), Association for Computing Machinery, Hong Kong, China, 111–118.

Digital Library

[17]

Aakar Gupta, Cheng Ji, Hui-Shyong Yeo, Aaron Quigley and Daniel Vogel. 2019. RotoSwype: word-gesture typing using a ring. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19), Association for Computing Machinery, Paper 14.

Digital Library

[18]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9 (8). 1735-1780.

[19]

Per-Ola Kristensson and Shumin Zhai. 2004. SHARK 2: a large vocabulary shorthand writing system for pen-based computers. In Proceedings of the 17th annual ACM symposium on User interface software and technology (UIST’04), ACM, 43-52.

Digital Library

[20]

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.

[21]

I Scott MacKenzie and Shawn X Zhang. 1999. The design and evaluation of a high-performance soft keyboard. In Proceedings of the SIGCHI conference on Human factors in computing systems (CHI’99), 25-31.

Digital Library

[22]

Anders Markussen, Mikkel Rønne Jakobsen and Kasper Hornbæk. 2014. Vulture: a mid-air word-gesture keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14), ACM, 1073-1082.

Digital Library

[23]

Mehryar Mohri. 1997. Finite-state transducers in language and speech processing. Computational linguistics, 23 (2). 269-311.

[24]

Tom Ouyang, David Rybach, Françoise Beaufays and Michael Riley. 2017. Mobile keyboard input decoding with finite-state transducers. arXiv preprint arXiv:1704.03987.

[25]

Kseniia Palin, Anna Maria Feit, Sunjun Kim, Per Ola Kristensson and Antti Oulasvirta. 2019. How do people type on mobile devices? Observations from a study with 37,000 volunteers. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI’19), 1-12.

[26]

Philip Quinn and Shumin Zhai. 2016. A cost-benefit study of text entry suggestion interaction. In Proceedings of the 2016 CHI conference on human factors in computing systems (CHI’16), 83-88.

Digital Library

[27]

Philip Quinn and Shumin Zhai. 2018. Modeling Gesture-Typing Movements. Human-Computer Interaction, 33 (3). 234-280.

[28]

Sascha Rothe, Shashi Narayan and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics 8. 264-280.

[29]

Emanuel Todorov and Michael I Jordan. 1998. Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. Journal of Neurophysiology, 80 (2). 696-714.

[30]

Keith Vertanen, Crystal Fletcher, Dylan Gaines, Jacob Gould and Per Ola Kristensson. 2018. The impact of word, multiple word, and sentence input on virtual keyboard decoding performance. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI’18), 1-12.

Digital Library

[31]

Keith Vertanen, Dylan Gaines, Crystal Fletcher, Alex M Stanage, Robbie Watling and Per Ola Kristensson. 2019. VelociWatch: Designing and evaluating a virtual keyboard for the input of challenging text. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19), 1-14.

Digital Library

[32]

Keith Vertanen, Haythem Memmi, Justin Emge, Shyam Reyal and Per Ola Kristensson. 2015. VelociTap: Investigating fast mobile text entry using sentence-based decoding of touchscreen keyboard input. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15), 659-668.

Digital Library

[33]

Thomas Wolf, Julien Chaumond, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison and Sam Shleifer. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP System Demonstration’20), 38-45.

[34]

Hui-Shyong Yeo, Xiao-Shen Phang, Steven J. Castellucci, Per Ola Kristensson and Aaron Quigley. 2017. Investigating tilt-based gesture keyboard entry for single-handed text entry on large devices. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17), Association for Computing Machinery, 4194–4202.

Digital Library

[35]

Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo and Yuanchun Shi. 2017. Tap, dwell or gesture? Exploring head-based text entry techniques for HMDs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17), Association for Computing Machinery, 4479–4488.

Digital Library

[36]

Shumin Zhai and Per-Ola Kristensson. 2003. Shorthand writing on stylus keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’03), Association for Computing Machinery, Ft. Lauderdale, Florida, USA, 97–104.

Digital Library

[37]

Shumin Zhai and Per Ola Kristensson. 2012. The word-gesture keyboard: reimagining keyboard interaction. Commun. ACM, 55 (9). 91–101. 10.1145/2330667.2330689

Digital Library

[38]

Shumin Zhai, Per Ola Kristensson, Pengjun Gong, Michael Greiner, Shilei Allen Peng, Liang Mico Liu and Anthony Dunnigan. 2009. Shapewriter on the iphone: from the laboratory to the real world. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA’09), Association for Computing Machinery, 2667–2670.

Digital Library

[39]

Mingrui Ray Zhang and Shumin Zhai. 2021. PhraseFlow: Designs and empirical studies of phrase-level input. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI’21), 1-13.

Digital Library

[40]

Xiang Zhang, Junbo Zhao and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (NeurIPS’15), MIT Press, Montreal, Canada, 649–657.

Digital Library

[41]

Suwen Zhu, Jingjie Zheng, Shumin Zhai and Xiaojun Bi. 2019. i'sFree: Eyes-free gesture typing via a touch-enabled remote control. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI’19), Association for Computing Machinery, Paper 448

Digital Library

Cited By

Xu ZCai SVarma T MVenugopalan SZhai S(2024)SkipWriter: LLM-Powered Abbreviated Writing on TabletsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676423(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676423
Shen JBoldu RKalla AGlueck MSurale HKarlson A(2024)RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction FrameworkIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345617930:11(7441-7451)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456179

Recommendations

Why Is Gesture Typing Promising for Older Adults?: Comparing Gesture and Tap Typing Behavior of Older with Young Adults
ASSETS '18: Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility

Gesture typing has been a widely adopted text entry method on touchscreen devices. We have conducted a study to understand whether older adults could gesture type, how they type, what are the strengths and weaknesses of gesture typing, and how to further ...
HGaze Typing: Head-Gesture Assisted Gaze Typing
ETRA '21 Full Papers: ACM Symposium on Eye Tracking Research and Applications

This paper introduces a bi-modal typing interface, HGaze Typing, which combines the simplicity of head gestures with the speed of gaze inputs to provide efficient and comfortable dwell-free text entry. HGaze Typing uses gaze path information to compute ...
Comparison of performance of enhanced morpheme-based language model with different word-based language models for improving the performance of Tamil speech recognition system

This paper describes a new technique of language modeling for a highly inflectional Dravidian language, Tamil. It aims to alleviate the main problems encountered in processing of Tamil language, like enormous vocabulary growth caused by the large number ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

October 2022

1363 pages

ISBN:9781450393201

DOI:10.1145/3526113

Editors:
Maneesh Agrawala
Stanford University, USA
,
Jacob O. Wobbrock
University of Washington, USA
,
Eytan Adar
University of Michigan, USA
,
Vidya Setlur
Tableau Research, USA

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UIST '22

Sponsor:

UIST '22: The 35th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 2, 2022

OR, Bend, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
599
Total Downloads

Downloads (Last 12 months)115
Downloads (Last 6 weeks)13

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu ZCai SVarma T MVenugopalan SZhai S(2024)SkipWriter: LLM-Powered Abbreviated Writing on TabletsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676423(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676423
Shen JBoldu RKalla AGlueck MSurale HKarlson A(2024)RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction FrameworkIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345617930:11(7441-7451)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456179

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents