More Web Proxy on the site http://driver.im/

research-article

Evaluating new requirements to pen-centric intelligent user interface based on end-to-end mathematical expressions recognition

Authors:

Dmytro Zhelezniakov,

Anastasiia Cherneha,

Viktor Zaytsev,

Tetiana Ignatova,

Olga Radyvonenko,

Oleg YakovchukAuthors Info & Claims

IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces

Pages 212 - 220

https://doi.org/10.1145/3377325.3377482

Published: 17 March 2020 Publication History

Abstract

Advances in deep learning end-to-end recognition systems allow moving forward intelligent user interfaces. At the same time, they lead to new implications and restrictions in the UI design. In this paper, we discuss new requirements for the pen-centric intelligent user interface for operating with mathematical expressions. We argue that the following features are necessary for such an interface: use of variables (both automatically assigned and defined by the user), quick editing, and connected writing support. We demonstrate and evaluate features in SMath - a solution for iterative input and automatic calculation of handwritten mathematical expressions. It allows pen-based input and editing of two-dimensional mathematical expressions in various scenarios with a possibility to reuse the results of previous calculations. Through the evaluation study, we justify the proposed requirements from the point of user needs and preferences. In addition to the automatic metrics that were collected during test tasks execution using the application, we gathered and summarized users' feedback, which can be further used to formulate functional requirements and design the user interface. We also provide the evaluation of the recognition performance of the system on an open-source dataset of handwritten mathematical expressions.

References

[1]

2019. MathBrush. https://apps.apple.com/ca/app/mathbrush/id578957934 Accessed: 2019-12-14.

[2]

2019. Microsoft Math Solver. https://math.microsoft.com/ Accessed: 2019-12-08.

[3]

2019. MyScript Calculator 2. https://www.myscript.com/calculator Accessed: 2019-12-08.

[4]

Francisco Álvaro, Joan-Andreu Sánchez, and José-Miguel Benedí. 2014. Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recognition Letters 35 (2014), 58 -- 67. Frontiers in Handwriting Processing.

Digital Library

[5]

Jared N. Bott. 2016. The WOZ Recognizer: A Tool For Understanding User Perceptions of Sketch-Based Interfaces. Ph.D. Dissertation. University of Central Florida. https://stars.library.ucf.edu/etd/4884

[6]

John Brooke. 1996. "SUS-A quick and dirty usability scale." Usability evaluation in industry. CRC Press. https://www.crcpress.com/product/isbn/9780748404605 ISBN: 9780748404605.

[7]

Kam-Fai Chan and Dit-Yan Yeung. 2001. Pencalc: A novel application of online mathematical expression recognition technology. In Proceedings of Sixth International Conference on Document Analysis and Recognition. IEEE, 774--778.

[8]

Andreu Comajuncosas, Marcos Faundez-Zanuy, Jordi SolÃl'-Casals, and Marta Portero-Tresserra. 2018. Preliminary Study on Implications of Cursive Handwriting Learning in Schools. 339--344.

[9]

A. Graves, M. Liwicki, S. Fernández, R. Bertolami, H. Bunke, and J. Schmidhuber. 2009. A Novel Connectionist System for Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 5 (May 2009), 855--868.

Digital Library

[10]

Eric Gross, Safwan Wshah, Isaiah Simmons, and Gary Skinner. 2015. A Handwriting Recognition System for the Classroom. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (LAK '15). ACM, New York, NY, USA, 218--222.

Digital Library

[11]

Jiseong Gu and Geehyuk Lee. 2016. In-Place-Ink: Toward More Direct Handwriting Interfaces. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (ISS '16). ACM, New York, NY, USA, 67--76.

Digital Library

[12]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload 1, 3 (1988), 139--183.

[13]

Joseph J La Viola Jr and Robert C Zeleznik. 2004. MathPad 2: a system for the creation and exploration of mathematical sketches. ACM Transactions on Graphics (TOG) 23, 3 (2004), 432--440.

Digital Library

[14]

A. D. Le and M. Nakagawa. 2017. Training an End-to-End System for Handwritten Mathematical Expression Recognition by Generated Patterns. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01. 1056--1061.

[15]

Daniel Lückehe, Nicole Mühlpforte, Alfred O. Effenberg, and Gabriele von Voigt. 2018. Multi-Stage Deep Learning for Context-Free Handwriting Recognition. In Proceedings of the Conference "Lernen, Wissen, Daten, Analysen", LWDA 2018, Mannheim, Germany, August 22--24, 2018. 239--250. http://ceur-ws.org/Vol-2191/paper28.pdf

[16]

Mahshad Mahdavi, Richard Zanibbi, Harold Mouchère, Christian Viard-Gaudin, and Utpal Garain. 2019. ICDAR 2019 CROHME + TFD: Competition on Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection. In 2019 International Conference on Document Analysis (ICDAR). https://www.cs.rit.edu/~rlaz/files/CROHME+TFD%E2%80%932019.pdf

[17]

S. Medjkoune, H. Mouchere, S. Petitrenaud, and C. Viard-Gaudin. 2017. Combining Speech and Handwriting Modalities for Mathematical Expression Recognition. IEEE Transactions on Human-Machine Systems 47, 2 (April 2017), 259--272.

[18]

Alexandra Mendes, Roland Backhouse, and Joao F. Ferreira. 2014. Structure Editing of Handwritten Mathematics: Improving the Computer Support for the Calculational Method. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (ITS '14). ACM, New York, NY, USA, 139--148.

Digital Library

[19]

Dan Meyer. 2016. The Future Of Handwriting Recognition & Adaptive Feedback In Math Education. https://blog.mrmeyer.com/2016/the-future-of-handwriting-recognition-adaptive-feedback-in-math-education/ Accessed: 2019-09-29.

[20]

Erik G. Miller and Paul A. Viola. 1998. Ambiguity and Constraint in Mathematical Expression Recognition. In Proceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence (AAAI '98/IAAI '98). American Association for Artificial Intelligence, Menlo Park, CA, USA, 784--791. http://dl.acm.org/citation.cfm?id=295240.295805

[21]

Harold Mouchère, Richard Zanibbi, Utpal Garain, and Christian Viard-Gaudin. 2016. Advancing the State of the Art for Handwritten Math Recognition: The CROHME Competitions, 2011--2014. Int. J. Doc. Anal. Recognit. 19, 2 (June 2016), 173--189.

Digital Library

[22]

Francisco Álvaro Muñoz. 2015. Mathematical Expression Recognition based on Probabilistic Grammars. Ph.D. Dissertation. Universitat Politécnica de Valéncia. https://pdfs.semanticscholar.org/5605/a2d4196c8388a0d4bbf03cfbb3f6d847cee4.pdf

[23]

Theresa O'Connell, Chuanjun Li, Timothy S. Miller, Robert C. Zeleznik, and Joseph J. LaViola Jr. 2009. A Usability Evaluation of AlgoSketch: A Pen-Based Application for Mathematics. Sketch-Based Interfaces and Modeling 2009 - ACM SIGGRAPH/Eurographics Symposium Proceedings, 149--158.

[24]

Tony D. Ryff. 2018. The use of cursive writing in a digital age: a mixed-methods analysis of the differences between modes of transcription in private schools. Ph.D. Dissertation. Northwest Nazarene University. http://repository.mnu.edu/sites/default/files/TonyRyffFINALDISSERTATION.pdf

[25]

Steve Smithies, Kevin Novins, and James Arvo. 1999. A Handwriting-based Equation Editor. In Proceedings of the 1999 Conference on Graphics Interface '99. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 84--91. http://dl.acm.org/citation.cfm?id=351631.351660

Digital Library

[26]

Ernesto Tapia and Raul Rojas. 2005. Recognition of on-line handwritten mathematical expressions in the e-chalk system-an extension. In Eighth International Conference on Document Analysis and Recognition (ICDAR'05). IEEE, 1206--1210.

Digital Library

[27]

Eugene M. Taranta and Joseph J. LaViola, Jr. 2015. Math Boxes: A Pen-Based User Interface for Writing Difficult Mathematical Expressions. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI '15). ACM, New York, NY, USA, 87--96.

Digital Library

[28]

Harold Thimbleby. 1995. A New Calculator and Why it is Necessary. Comput. J. 38, 6 (01 1995), 418--433. arXiv:http://oup.prod.sis.lan/comjnl/article-pdf/38/6/418/982950/38-6-418.pdf

[29]

William Thimbleby. 2004. A Novel Pen-based Calculator and Its Evaluation. In Proceedings of the Third Nordic Conference on Human-computer Interaction (NordiCHI '04). ACM, New York, NY, USA, 445--448.

Digital Library

[30]

Will Thimbleby and Harold Thimbleby. 2005. A novel gesture-based calculator and its design principles. In Proceedings of 19th BCS HCI Conference, Vol. 02. 27--32.

[31]

Robert Zeleznik, Timothy Miller, Chuanjun Li, and Joseph J. LaViola. 2008. Math-Paper: Mathematical Sketching with Fluid Support for Interactive Computation. In Smart Graphics, Andreas Butz, Brian Fisher, Antonio Krüger, Patrick Olivier, and Marc Christie (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 20--32.

[32]

Jianshu Zhang, Jun Du, Shiliang Zhang, Dan Liu, Yulong Hu, Jinshui Hu, Si Wei, and Lirong Dai. 2017. Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognition 71 (2017), 196 -- 206.

[33]

T. Zhang, H. Mouchère, and C. Viard-Gaudin. 2017. Tree-Based BLSTM for Mathematical Expression Recognition. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01. 914--919.

[34]

Wei Zhang, Zhiqiang Bai, and Yuesheng Zhu. 2019. An Improved Approach Based on CNN-RNNs for Mathematical Expression Recognition. In Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal Processing (ICMSSP 2019). ACM, New York, NY, USA, 57--61.

Digital Library

[35]

Dmitry Zhelezniakov, Viktor Zaytsev, and Olga Radyvonenko. 2019. Acceleration of Online Recognition of 2D Sequences Using Deep Bidirectional LSTM and Dynamic Programming. 438--449.

Cited By

Drivet A(2024)Smartphone and STEMUsing STEM-Focused Teacher Preparation Programs to Reimagine Elementary Education10.4018/978-1-6684-5939-3.ch009(196-230)Online publication date: 7-Feb-2024
https://doi.org/10.4018/978-1-6684-5939-3.ch009
Kang WHuang JTong QFu QTian FDai G(2024)MathAssist: A Handwritten Mathematical Expression Autocomplete TechniqueProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645149(566-581)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645149
Shirai SNakahara TFukui T(2023)MathTOUCH Editor: Rich-text Editor for Math E-learning Using an Intelligent Math Input InterfaceJournal of Information Processing10.2197/ipsjjip.31.77531(775-785)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.775
Show More Cited By

Index Terms

Evaluating new requirements to pen-centric intelligent user interface based on end-to-end mathematical expressions recognition
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User centered design
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods

Recommendations

A Pen-Based Prosodic User Interface for Schoolchildren

A prosodic user interface is defined as an user interface that can deal with not only what is entered by the user but also how it is entered. A pen-based user interface provides more prosodic information than a mouse-based graphical user interface. The ...
ICDAR 2023 Competition on Recognition of Multi-line Handwritten Mathematical Expressions
Document Analysis and Recognition - ICDAR 2023
Abstract
Mathematical expressions play an essential role in scientific documents and are critical for describing problems and theories in various fields, such as mathematics and physics. Consequently, the automatic recognition of handwritten mathematical ...
Handwritten Urdu character recognition using one-dimensional BLSTM classifier

The recognition of cursive script is regarded as a subtle task in optical character recognition due to its varied representation. Every cursive script has different nature and associated challenges. As Urdu is one of cursive language that is derived ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IUI '20: Proceedings of the 25th International Conference on Intelligent User Interfaces

March 2020

607 pages

ISBN:9781450371186

DOI:10.1145/3377325

General Chairs:
Fabio Paternò,
Nuria Oliver,
Program Chairs:
Cristina Conati,
Lucio Davide Spano,
Nava Tintarev

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IUI '20

Sponsor:

IUI '20: 25th International Conference on Intelligent User Interfaces

March 17 - 20, 2020

Cagliari, Italy

Acceptance Rates

Overall Acceptance Rate 746 of 2,811 submissions, 27%

Upcoming Conference

IUI '25

Sponsor:
sigai
sigai

30th International Conference on Intelligent User Interfaces

March 24 - 27, 2025

Cagliari , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
293
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Drivet A(2024)Smartphone and STEMUsing STEM-Focused Teacher Preparation Programs to Reimagine Elementary Education10.4018/978-1-6684-5939-3.ch009(196-230)Online publication date: 7-Feb-2024
https://doi.org/10.4018/978-1-6684-5939-3.ch009
Kang WHuang JTong QFu QTian FDai G(2024)MathAssist: A Handwritten Mathematical Expression Autocomplete TechniqueProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645149(566-581)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645149
Shirai SNakahara TFukui T(2023)MathTOUCH Editor: Rich-text Editor for Math E-learning Using an Intelligent Math Input InterfaceJournal of Information Processing10.2197/ipsjjip.31.77531(775-785)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.775
Sakshi Kukreja V(2023)Machine learning and non-machine learning methods in mathematical recognition systems: Two decades’ systematic literature reviewMultimedia Tools and Applications10.1007/s11042-023-16356-z83:9(27831-27900)Online publication date: 23-Aug-2023
https://doi.org/10.1007/s11042-023-16356-z
Zaytsev VZhelezniakov DCherneha ARadyvonenko O(2023)A Study on the Usability of Handwriting Assistant for Smartphone’s Lock ScreenDesign, Operation and Evaluation of Mobile Communications10.1007/978-3-031-35921-7_8(106-123)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35921-7_8
Sakshi Kukreja V(2022)A dive in white and grey shades of ML and non-ML literature: a multivocal analysis of mathematical expressionsArtificial Intelligence Review10.1007/s10462-022-10330-156:7(7047-7135)Online publication date: 6-Dec-2022
https://doi.org/10.1007/s10462-022-10330-1
Kaluarachchi TReis ANanayakkara S(2021)A Review of Recent Deep Learning Approaches in Human-Centered Machine LearningSensors10.3390/s2107251421:7(2514)Online publication date: 3-Apr-2021
https://doi.org/10.3390/s21072514
Zhelezniakov DZaytsev VRadyvonenko O(2021)Online Handwritten Mathematical Expression Recognition and Applications: A SurveyIEEE Access10.1109/ACCESS.2021.30634139(38352-38373)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3063413
Yakovchuk OCherneha AZhelezniakov DZaytsev V(2020)Methods for Lines and Matrices Segmentation in RNN-based Online Handwriting Mathematical Expression Recognition Systems2020 IEEE Third International Conference on Data Stream Mining & Processing (DSMP)10.1109/DSMP47368.2020.9204273(255-261)Online publication date: Aug-2020
https://doi.org/10.1109/DSMP47368.2020.9204273

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents