More Web Proxy on the site http://driver.im/

research-article

Public Access

A Rapid Prototyping Approach to Synthetic Data Generation for Improved 2D Gesture Recognition

Authors:

Eugene M. Taranta, II,

Mehran Maghoumi,

Corey R. Pittman,

Joseph J. LaViola, Jr.Authors Info & Claims

UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

Pages 873 - 885

https://doi.org/10.1145/2984511.2984525

Published: 16 October 2016 Publication History

Abstract

Training gesture recognizers with synthetic data generated from real gestures is a well known and powerful technique that can significantly improve recognition accuracy. In this paper we introduce a novel technique called gesture path stochastic resampling (GPSR) that is computationally efficient, has minimal coding overhead, and yet despite its simplicity is able to achieve higher accuracy than competitive, state-of-the-art approaches. GPSR generates synthetic samples by lengthening and shortening gesture subpaths within a given sample to produce realistic variations of the input via a process of nonuniform resampling. As such, GPSR is an appropriate rapid prototyping technique where ease of use, understandability, and efficiency are key. Further, through an extensive evaluation, we show that accuracy significantly improves when gesture recognizers are trained with GPSR synthetic samples. In some cases, mean recognition errors are reduced by more than 70%, and in most cases, GPSR outperforms two other evaluated state-of-the-art methods.

Supplementary Material

MP4 File (p873-taranta.mp4)

Download
196.69 MB

References

[1]

Anthony, L., and Wobbrock, J. O. A lightweight multistroke recognizer for user interface prototypes. In Proceedings of Graphics Interface 2010, GI '10, Canadian Information Processing Society (Toronto, Ont., Canada, Canada, 2010), 245--252.

Digital Library

[2]

Anthony, L., and Wobbrock, J. O. $n-protractor: A fast and accurate multistroke recognizer. In Proceedings of Graphics Interface 2012, GI '12, Canadian Information Processing Society (Toronto, Ont., Canada, Canada, 2012), 117--120.

Digital Library

[3]

Appert, C., and Zhai, S. Using strokes as command shortcuts: Cognitive bene'ts and toolkit support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, ACM (New York, NY, USA, 2009), 2289--2298.

Digital Library

[4]

Awal, A.-M., Mouchere, H., and Viard-Gaudin, C. Towards handwritten mathematical expression recognition. In Document Analysis and Recognition, 2009. ICDAR '09. 10th International Conference on (July 2009), 1046--1050.

Digital Library

[5]

Blagojevic, R., Chang, S. H.-H., and Plimmer, B. The power of automatic feature selection: Rubine on steroids. In Proceedings of the Seventh Sketch-Based Interfaces and Modeling Symposium, SBIM '10, Eurographics Association (Aire-la-Ville, Switzerland, Switzerland, 2010), 79--86.

Digital Library

[6]

Cano, J., Perez-Cortes, J.-C., Arlandis, J., and Llobet, R. Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshops SSPR 2002 and SPR 2002 Windsor, Ontario, Canada, August 6'9, 2002 Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, 2002, ch. Training Set Expansion in Handwritten Character Recognition, 548--556.

Digital Library

[7]

Cao, X., and Zhai, S. Modeling human performance of pen stroke gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '07, ACM (New York, NY, USA, 2007), 1495--1504.

Digital Library

[8]

Caramiaux, B., Montecchio, N., Tanaka, A., and Bevilacqua, F. Adaptive gesture recognition with variation estimation for interactive systems. ACM Trans. Interact. Intell. Syst. 4, 4 (Dec. 2014), 18:1--18:34.

Digital Library

[9]

Davila, K., Ludi, S., and Zanibbi, R. Using o'-line features and synthetic data for on-line handwritten math symbol recognition. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on, IEEE (2014), 323--328.

[10]

Dinges, L., Elzobi, M., Al-Hamadi, A., and Aghbari, Z. A. Image Processing and Communications Challenges 3. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011, ch. Synthizing Handwritten Arabic Text Using Active Shape Models, 401--408.

[11]

Duda, R. O., Hart, P. E., and Stork, D. G. Pattern Classi'cation (2nd Edition). Wiley-Interscience, 2001.

Digital Library

[12]

Elanwar, R. I. The state of the art in handwriting synthesis. In 2nd International Conference on New Paradigms in Electronics & information Technology (peit'013), Luxor, Egypt (2013).

[13]

Farooq, F., Jose, D., and Govindaraju, V. Phrase-based correction model for improving handwriting recognition accuracies. Pattern Recogn. 42, 12 (Dec. 2009), 3271--3277.

Digital Library

[14]

Fischer, A., Plamondon, R., O'Reilly, C., and Savaria, Y. Neuromuscular representation and synthetic generation of handwritten whiteboard notes. In Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on (Sept 2014), 222--227.

[15]

Fischer, A., Visani, M., Kieu, V. C., and Suen, C. Y. Generation of learning samples for historical handwriting recognition using image degradation. In Proceedings of the 2Nd International Workshop on Historical Document Imaging and Processing, HIP '13, ACM (New York, NY, USA, 2013), 73--79.

Digital Library

[16]

Galbally, J., Fierrez, J., Martinez-Diaz, M., and Ortega-Garcia, J. Synthetic generation of handwritten signatures based on spectral analysis. In SPIE Defense, Security, and Sensing, International Society for Optics and Photonics (2009), 730629--730629.

[17]

Gatos, B., Konidaris, T., Ntzios, K., Pratikakis, I., and Perantonis, S. J. A segmentation-free approach for keyword search in historical typewritten documents. In Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR '05, IEEE Computer Society (Washington, DC, USA, 2005), 54--58.

Digital Library

[18]

Ha, T. M., and Bunke, H. O'-line, handwritten numeral recognition by perturbation method. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 5 (May 1997), 535--539.

Digital Library

[19]

Helmers, M., and Bunke, H. Generation and use of synthetic training data in cursive handwriting recognition. In Pattern Recognition and Image Analysis. Springer, 2003, 336--345.

[20]

Herold, J., and Stahovich, T. F. Speedseg: A technique for segmenting pen strokes using pen speed. Computers & Graphics 35, 2 (2011), 250 ' 264. Virtual Reality in BrazilVisual Computing in Biology and MedicineSemantic 3D media and contentCultural Heritage.

Digital Library

[21]

Herold, J., and Stahovich, T. F. The 1¢ recognizer: A fast, accurate, and easy-to-implement handwritten gesture recognition technique. In Proceedings of the International Symposium on Sketch-Based Interfaces and Modeling, SBIM '12, Eurographics Association (Aire-la-Ville, Switzerland, Switzerland, 2012), 39--46.

Digital Library

[22]

II, E. M. T., Vargas, A. N., and Jr., J. J. L. Streamlined and accurate gesture recognition with penny pincher. Computers & Graphics 55 (2016), 130--142.

Digital Library

[23]

Kutner, M. H., Nachtsheim, C. J., Neter, J., and Li, W. Applied linear statistical models, vol. 5. McGraw-Hill Irwin New York, 2005.

[24]

Lee, D.-H., and Cho, H.-G. A new synthesizing method for handwriting korean scripts. International Journal of Pattern Recognition and Arti'cial Intelligence 12, 01 (1998), 45--61.

[25]

Leiva, L. A., Martín-Albo, D., and Plamondon, R. Gestures À go go: Authoring synthetic human-like stroke gestures using the kinematic theory of rapid movements. ACM Trans. Intell. Syst. Technol. 7, 2 (Nov. 2015), 15:1--15:29.

Digital Library

[26]

Li, Y. Protractor: A fast and accurate gesture recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, ACM (New York, NY, USA, 2010), 2169--2172.

Digital Library

[27]

Lü, H., Fogarty, J. A., and Li, Y. Gesture script: Recognizing gestures and their structure using rendering scripts and interactively trained parts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '14, ACM (New York, NY, USA, 2014), 1685--1694.

Digital Library

[28]

Lundin, E., Kvarnström, H., and Jonsson, E. A synthetic fraud data generation methodology. In Proceedings of the 4th International Conference on Information and Communications Security, ICICS '02, Springer-Verlag (London, UK, UK, 2002), 265--277.

Digital Library

[29]

MacLean, S., Tausky, D., Labahn, G., Lank, E., and Marzouk, M. Tools for the e cient generation of hand-drawn corpora based on context-free grammars. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, SBIM '09, ACM (New York, NY, USA, 2009), 125--132.

Digital Library

[30]

Martín-Albo, D., Plamondon, R., and Vidal, E. Improving sigma-lognormal parameter extraction. In Document Analysis and Recognition (ICDAR), 2015 13th International Conference on (Aug 2015), 286--290.

Digital Library

[31]

Navaratnam, R., Fitzgibbon, A. W., and Cipolla, R. The joint manifold model for semi-supervised multi-valued regression. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on (Oct 2007), 1--8.

[32]

Perlin, K. An image synthesizer. In Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '85, ACM (New York, NY, USA, 1985), 287--296.

Digital Library

[33]

Pittman, C., Taranta II, E. M., and LaViola, Jr., J. J. A $-family friendly approach to prototype selection. In Proceedings of the 21st International Conference on Intelligent User Interfaces, IUI '16, ACM (New York, NY, USA, 2016), 370--374.

Digital Library

[34]

Plamondon, R. A kinematic theory of rapid human movements. Biological cybernetics 72, 4 (1995), 295--307.

Digital Library

[35]

Plamondon, R., and Djioua, M. A multi-level representation paradigm for handwriting stroke generation. Human movement science 25, 4 (2006), 586--607.

[36]

Rodriguez-Serrano, J. A., and Perronnin, F. Synthesizing queries for handwritten word image retrieval. Pattern Recognition 45, 9 (2012), 3270--3276. Best Papers of Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA'2011).

Digital Library

[37]

Rowley, H. A., Goyal, M., and Bennett, J. The e'ect of large training set sizes on online japanese kanji and english cursive recognizers. In Frontiers in Handwriting Recognition, 2002. Proceedings. Eighth International Workshop on (2002), 36--40.

Digital Library

[38]

Rubine, D. Specifying gestures by example. SIGGRAPH Computer Graphics 25, 4 (July 1991), 329--337.

Digital Library

[39]

Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. Real-time human pose recognition in parts from single depth images. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '11, IEEE Computer Society (Washington, DC, USA, 2011), 1297--1304.

Digital Library

[40]

Thomas, A. O., Rusu, A., and Govindaraju, V. Synthetic handwritten captchas. Pattern Recogn. 42, 12 (Dec. 2009), 3365--3373.

Digital Library

[41]

Varga, T., and Bunke, H. Generation of synthetic training data for an hmm-based handwriting recognition system. In Document Analysis and Recognition, 2003. Proceedings. Seventh International Conference on (Aug 2003), 618--622 vol.1.

Digital Library

[42]

Varga, T., and Bunke, H. Offline handwriting recognition using synthetic training data produced by means of a geometrical distortion model. International Journal of Pattern Recognition and Arti'cial Intelligence 18, (2004), 1285--1302.

[43]

Varga, T., Kilchhofer, D., and Bunke, H. Template-based synthetic handwriting generation for the training of recognition systems. In Proceedings of the 12th Conference of the International Graphonomics Society (2005), 206--211.

[44]

Vatavu, R.-D. The e'ect of sampling rate on the performance of template-based gesture recognizers. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI '11, ACM (New York, NY, USA, 2011), 271--278.

Digital Library

[45]

Vatavu, R.-D., Anthony, L., and Wobbrock, J. O. Gestures as point clouds: A $p recognizer for user interface prototypes. In Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI '12, ACM (New York, NY, USA, 2012), 273--280.

Digital Library

[46]

Vatavu, R.-D., Anthony, L., and Wobbrock, J. O. Relative accuracy measures for stroke gestures. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI '13, ACM (New York, NY, USA, 2013), 279--286.

Digital Library

[47]

Vatavu, R.-D., Vogel, D., Casiez, G., and Grisoni, L. Estimating the perceived di culty of pen gestures. In Proceedings of the 13th IFIP TC 13 International Conference on Human-computer Interaction - Volume Part II, INTERACT'11, Springer-Verlag (Berlin, Heidelberg, 2011), 89--106.

Digital Library

[48]

Velek, O., and Nakagawa, M. Document Analysis Systems V: 5th International Workshop, DAS 2002 Princeton, NJ, USA, August 19'21, 2002 Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, 2002, ch. The Impact of Large Training Sets on the Recognition Rate of O'-line Japanese Kanji Character Classi'ers, 106--110.

Digital Library

[49]

Wobbrock, J. O., Findlater, L., Gergle, D., and Higgins, J. J. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (New York, NY, USA, 2011), 143--146.

Digital Library

[50]

Wobbrock, J. O., Wilson, A. D., and Li, Y. Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST '07, ACM (New York, NY, USA, 2007), 159--168.

Digital Library

Cited By

Bachert MHesenius M(2024)Towards a Framework for Evaluating Synthetic Surface GesturesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661327(22-30)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661327
Lu HXu SZhao SHu XMa RHu B(2024)EPIC: Emotion Perception by Spatio-Temporal Interaction Context of GaitIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2022.323359728:5(2592-2601)Online publication date: May-2024
https://doi.org/10.1109/JBHI.2022.3233597
Gomaa AZitt RReyes GKrüger A(2024)SynthoGestures: A Multi-Camera Framework for Generating Synthetic Dynamic Hand Gestures for Enhanced Vehicle Interaction2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588662(3297-3303)Online publication date: 2-Jun-2024
https://doi.org/10.1109/IV55156.2024.10588662
Show More Cited By

Index Terms

A Rapid Prototyping Approach to Synthetic Data Generation for Improved 2D Gesture Recognition
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)

Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Exploiting Speech/Gesture Co-occurrence for Improving Continuous Gesture Recognition in Weather Narration
FG '00: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000

In order to incorporate naturalness in the design of Human Computer Interfaces (HCI), it is desirable to develop recognition techniques capable of handling continuous natural gesture and speech inputs. Though many different researchers have reported ...
Multi-User Gesture Recognition Using WiFi
MobiSys '18: Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services

WiFi based gesture recognition has received significant attention over the past few years. However, the key limitation of prior WiFi based gesture recognition systems is that they cannot recognize the gestures of multiple users performing them ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '16: Proceedings of the 29th Annual Symposium on User Interface Software and Technology

October 2016

908 pages

ISBN:9781450341899

DOI:10.1145/2984511

General Chairs:
Jun Rekimoto
The University of Tokyo
,
Takeo Igarashi
The University of Tokyo
,
Program Chairs:
Jacob O. Wobbrock
University of Washington
,
Daniel Avrahami
FXPAL

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

UIST '16

Sponsor:

UIST '16: The 29th Annual ACM Symposium on User Interface Software and Technology

October 16 - 19, 2016

Tokyo, Japan

Acceptance Rates

UIST '16 Paper Acceptance Rate 79 of 384 submissions, 21%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
689
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)5

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bachert MHesenius M(2024)Towards a Framework for Evaluating Synthetic Surface GesturesCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661327(22-30)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661327
Lu HXu SZhao SHu XMa RHu B(2024)EPIC: Emotion Perception by Spatio-Temporal Interaction Context of GaitIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2022.323359728:5(2592-2601)Online publication date: May-2024
https://doi.org/10.1109/JBHI.2022.3233597
Gomaa AZitt RReyes GKrüger A(2024)SynthoGestures: A Multi-Camera Framework for Generating Synthetic Dynamic Hand Gestures for Enhanced Vehicle Interaction2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588662(3297-3303)Online publication date: 2-Jun-2024
https://doi.org/10.1109/IV55156.2024.10588662
Gomaa AZitt RReyes GKrüger A(2023)SynthoGestures: A Novel Framework for Synthetic Dynamic Hand Gesture Generation for Driving ScenariosAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3616635(1-3)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586182.3616635
Gomaa AZitt RReyes G(2023)Advancing Dynamic Hand Gesture Recognition in Driving Scenarios with Synthetic DataAdjunct Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications10.1145/3581961.3609889(1-6)Online publication date: 18-Sep-2023
https://dl.acm.org/doi/10.1145/3581961.3609889
Maslych MTaranta EAldilati MLaviola J(2023)Effective 2D Stroke-based Gesture Augmentation for RNNsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581358(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581358
Sluÿters AOusmer MRoselli PVanderdonckt J(2022)QuantumLeap, a Framework for Engineering Gestural User Interfaces based on the Leap Motion ControllerProceedings of the ACM on Human-Computer Interaction10.1145/35322116:EICS(1-47)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532211
Magrofuoco NRoselli PVanderdonckt J(2022)µV: An Articulation, Rotation, Scaling, and Translation Invariant (ARST) Multi-stroke Gesture RecognizerProceedings of the ACM on Human-Computer Interaction10.1145/35322006:EICS(1-25)Online publication date: 17-Jun-2022
https://dl.acm.org/doi/10.1145/3532200
Taranta EMaslych MGhamandi RLaViola J(2022)The Voight-Kampff Machine for Automatic Custom Gesture Rejection Threshold SelectionProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502000(1-15)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502000
Shen JDudley JMo GKristensson P(2022)Gesture Spotter: A Rapid Prototyping Tool for Key Gesture Spotting in Virtual and Augmented Reality ApplicationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320300428:11(3618-3628)Online publication date: Nov-2022
https://doi.org/10.1109/TVCG.2022.3203004
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents