short-paper

TouchML: A Machine Learning Toolkit for Modelling Spatial Touch Targeting Behaviour

Authors:

Daniel Buschek,

Florian AltAuthors Info & Claims

IUI '15: Proceedings of the 20th International Conference on Intelligent User Interfaces

Pages 110 - 114

https://doi.org/10.1145/2678025.2701381

Published: 18 March 2015 Publication History

Get Access

Abstract

Pointing tasks are commonly studied in HCI research, for example to evaluate and compare different interaction techniques or devices. A recent line of work has modelled user-specific touch behaviour with machine learning methods to reveal spatial targeting error patterns across the screen. These models can also be applied to improve accuracy of touchscreens and keyboards, and to recognise users and hand postures. However, no implementation of these techniques has been made publicly available yet, hindering broader use in research and practical deployments. Therefore, this paper presents a toolkit which implements such touch models for data analysis (Python), mobile applications (Java/Android), and the web (JavaScript). We demonstrate several applications, including hand posture recognition, on touch targeting data collected in a study with 24 participants. We consider different target types and hand postures, changing behaviour over time, and the influence of hand sizes.

References

[1]

Baldwin, T., and Chai, J. Towards Online Adaptation and Personalization of Key-Target Resizing for Mobile Devices. In Proceedings of the 17th International Conference on Intelligent User Interfaces (2012), 11--20.

Digital Library

Google Scholar

[2]

Bergstrom-Lehtovirta, J., and Oulasvirta, A. Modeling the Functional Area of the Thumb on Mobile Touchscreen Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2014), 1991--2000.

Digital Library

Google Scholar

[3]

Bi, X., Li, Y., and Zhai, S. FFitts Law: Modeling Finger Touch with Fitts' Law. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2013), 1363--1372.

Digital Library

Google Scholar

[4]

Buschek, D., Rogers, S., and Murray-Smith, R. User-Specific Touch Models in a Cross-Device Context. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services (2013), 382--391.

Digital Library

Google Scholar

[5]

Fitts, P. M. The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement. Journal of Experimental Psychology 47, 6 (1954), 381--391.

Crossref

Google Scholar

[6]

Henze, N., Rukzio, E., and Boll, S. 100,000,000 Taps: Analysis and Improvement of Touch Performance in the Large. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (2011), 133--142.

Digital Library

Google Scholar

[7]

Mock, P., Edelmann, J., Schilling, A., and Rosenstiel, W. User Identification Using Raw Sensor Data From Typing on Interactive Displays. In Proceedings of the 19th International Conference on Intelligent User Interfaces (2014), 67--72.

Digital Library

Google Scholar

[8]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12 (Nov. 2011), 2825--2830.

Digital Library

Google Scholar

[9]

Rasmussen, C. E., and Williams, C. K. I. Gaussian Processes for Machine Learning. The MIT Press, 2006.

Digital Library

Google Scholar

[10]

van der Walt, S., Colbert, S., and Varoquaux, G. The NumPy Array: A Structure for Efficient Numerical Computation. Computing in Science Engineering 13, 2 (March 2011), 22--30.

Digital Library

Google Scholar

[11]

Weir, D., Buschek, D., and Rogers, S. Sparse Selection of Training Data for Touch Correction Systems. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services (2013), 404--407.

Digital Library

Google Scholar

[12]

Weir, D., Pohl, H., Rogers, S., Vertanen, K., and Kristensson, P. Uncertain Text Entry on Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2014), 2307--2316.

Digital Library

Google Scholar

[13]

Weir, D., Rogers, S., Murray-Smith, R., and Löchtefeld, M. A User-Specific Machine Learning Approach for Improving Touch Accuracy on Mobile Devices. In Proceedings of the 25th annual Symposium on User Interface Software and Technology (2012), 465--476.

Digital Library

Google Scholar

[14]

Yin, Y., Ouyang, T. Y., Partridge, K., and Zhai, S. Making Touchscreen Keyboards Adaptive to Keys, Hand Postures, and Individuals - A Hierarchical Spatial Backoff Model Approach. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2013), 2775--2784.

Digital Library

Google Scholar

Cited By

View all

Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330
Lehmann FKornecki IBuschek DFeit A(2023)Typing Behavior is About More than Speed: Users' Strategies for Choosing Word Suggestions Despite Slower Typing RatesProceedings of the ACM on Human-Computer Interaction10.1145/36042767:MHCI(1-26)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604276
Li ZZhao MDas DZHAO HMa YLiu WBeaudouin-Lafon MWang FRamakrishnan IBi X(2022)Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517472(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517472
Show More Cited By

Index Terms

TouchML: A Machine Learning Toolkit for Modelling Spatial Touch Targeting Behaviour
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Touch screens

Recommendations

Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit
ITS '11: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces

Recent work in multi-touch tabletop interaction introduced many novel techniques that let people manipulate digital content through touch. Yet most only detect touch blobs. This ignores richer interactions that would be possible if we could identify (1) ...
A toolkit for analysis and prediction of touch targeting behaviour on mobile websites
EICS '15: Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems

Touch interaction on mobile devices suffers from several problems, such as the thumb's limited reach or the occlusion of targets by the finger. This leads to offsets between the user's intended touch location and the actual location sensed by the ...
Gaze and Touch Interaction on Tablets
UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

We explore how gaze can support touch interaction on tablets. When holding the device, the free thumb is normally limited in reach, but can provide an opportunity for indirect touch input. Here we propose gaze and touch input, where touches redirect to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

IUI '15: Proceedings of the 20th International Conference on Intelligent User Interfaces

March 2015

480 pages

ISBN:9781450333061

DOI:10.1145/2678025

General Chairs:
Oliver Brdiczka
Vectra Networks, Inc.
,
Polo Chau
Georgia Tech
,
Program Chairs:
Giuseppe Carenini
University of British Columbia
,
Shimei Pan
University of Maryland
,
Per Ola Kristensson
University of Cambridge

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

IUI'15

Sponsor:

IUI'15: IUI'15 20th International Conference on Intelligent User Interfaces

March 29 - April 1, 2015

Georgia, Atlanta, USA

Acceptance Rates

IUI '15 Paper Acceptance Rate 47 of 205 submissions, 23%;

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
405
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)4

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Streli PRichardson MBotros FMa SWang RHolz C(2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676330
Lehmann FKornecki IBuschek DFeit A(2023)Typing Behavior is About More than Speed: Users' Strategies for Choosing Word Suggestions Despite Slower Typing RatesProceedings of the ACM on Human-Computer Interaction10.1145/36042767:MHCI(1-26)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604276
Li ZZhao MDas DZHAO HMa YLiu WBeaudouin-Lafon MWang FRamakrishnan IBi X(2022)Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517472(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517472
van Berkel NMerritt TBruun ASkov M(2022)Tangible Self-Report Devices: Accuracy and Resolution of Participant InputProceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3490149.3501309(1-14)Online publication date: 13-Feb-2022
https://dl.acm.org/doi/10.1145/3490149.3501309
Zhao MHuang HLi ZLiu RCui WToshniwal KGoel AWang AZhao XRashidian SBaig FPhi KZhai SRamakrishnan IWang FBi X(2022)EyeSayCorrect: Eye Gaze and Voice Based Hands-free Text Correction for Mobile DevicesProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511103(470-482)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511103
Buschek DAlt F(2021)Building Adaptive Touch Interfaces—Case Study 6Intelligent Computing for Interactive System Design10.1145/3447404.3447426(379-406)Online publication date: 23-Feb-2021
https://dl.acm.org/doi/10.1145/3447404.3447426
Vertanen K(2021)Probabilistic Text Entry—Case Study 3Intelligent Computing for Interactive System Design10.1145/3447404.3447420(277-320)Online publication date: 23-Feb-2021
https://dl.acm.org/doi/10.1145/3447404.3447420
Prange SMecke LNguyen AKhamis MAlt F(2020)Don't Use Fingerprint, it's Raining!Proceedings of the 2020 International Conference on Advanced Visual Interfaces10.1145/3399715.3399823(1-5)Online publication date: 28-Sep-2020
https://dl.acm.org/doi/10.1145/3399715.3399823
Zhu SKim YZheng JLuo JQin RWang LFan XTian FBi XBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Using Bayes' Theorem for Command Input: Principle, Models, and ApplicationsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376771(1-15)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376771
Mecke LBuschek DKiermeier MPrange SAlt F(2019)Exploring intentional behaviour modifications for password typing on mobile touchscreen devicesProceedings of the Fifteenth USENIX Conference on Usable Privacy and Security10.5555/3361476.3361499(303-318)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.5555/3361476.3361499
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit

A toolkit for analysis and prediction of touch targeting behaviour on mobile websites

Gaze and Touch Interaction on Tablets