[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Exploiting on-the-fly interpretation to design technical documents in a mobile context

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Pen-based interaction is well adapted for writing down information in a mobile context. However, there is a lack of software taking advantage of this interaction process to design technical documents in constrained environments. This is because sketch interpretation is a complex research problem and good performances are required to design industrial software. The first contribution of this article is to show how “on-the-fly” interpretation is a viable approach to design robust and efficient sketch interpretation systems. With this process, the system interprets each stroke after its drawing and produces a visual feedback to the user. The interaction with the user is then strongly taken into account to get pertinent information and better interpret the document. Moreover, on-the-fly interpretation opens a large panel of possibilities to design accurate documents, such as contextual help, complex layout reproduction, etc. Of course, such a process has several requirements to be usable in practice, for example to avoid disturbing the user with inconsistent feedback. The second contribution of this paper is a set of strategies, based on grammatical modeling and pattern recognition, to face these needs. We present two industrial pen-based software that are based on the principles we present in this article. One of them, dedicated to architectural floor-plan design, as not been presented previously.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Almaksour A, Anquetil E, Quiniou S, Cheriet M (2010) Personalizable pen-based interface using lifelong learning. In: Proceedings of the international conference on frontiers in handwriting recognition (ICFHR’10)

    Google Scholar 

  2. Alvarado C, Davis R (2004) Sketchread: a multi-domain sketch recognition engine. In: Proceedings of the 17th annual ACM symposium on user interface software and technology (UIST’04), pp 23–32

    Chapter  Google Scholar 

  3. Anstice J, Bell T, Cockburn A, Setchell M (1996) The design of a pen-based musical input system. In: Proceedings of the sixth Australian conference on computer-human interaction (OzChi’96), pp 260–267

    Chapter  Google Scholar 

  4. Blostein D, Lank E, Rose A, Zanibbi R (2001) User interfaces for on-line diagram recognition. In: Selected papers from the fourth international workshop on graphics recognition algorithms and applications. LNCS, vol 2390. Springer, Berlin, pp 93–103

    Google Scholar 

  5. Caetano A, Goulart N, Fonseca M, Jorge J (2002) Javasketchit: Issues in sketching the look of user interfaces. In: Proceedings of the AAAI spring symposium on sketch understanding (SSSU’02)

    Google Scholar 

  6. Chen Q, Grundy J, Hosking J (2003) An e-whiteboard application to support early design-stage sketching of uml diagrams. In: Proceedings of the 2003 IEEE symposium on human centric computing languages and environments, pp 219–226

    Chapter  Google Scholar 

  7. Chok SS, Marriott K (2003) Automatic generation of intelligent diagram editors. ACM Trans Comput-Hum Interact 10(3):244–276

    Article  Google Scholar 

  8. Costagliola G, Deufemia V, Risi M (2005) Sketch grammars: A formalism for describing and recognizing diagrammatic sketch languages. In: Proceedings of the eighth international conference on document analysis and recognition (ICDAR’05), Seoul, Korea, pp 1226–1231

    Chapter  Google Scholar 

  9. Coyette A, Kieffer S, Vanderdonckt J (2007) Multi-fidelity prototyping of user interfaces. In: INTERACT 2007. LNCS, vol 4662. Springer, Berlin, pp 150–164. Part I

    Google Scholar 

  10. Donaldson AF, Williamson A (2005) Pen-based unput of uml activity diagrams for business process modelling. In: Proceedings of the first workshop on improving and assessing pen-based input techniques

    Google Scholar 

  11. Evodia (2010) http://www.evodia.fr

  12. Feng G, Viard-Gaudin C, Sun Z (2009) On-line hand-drawn electric circuit diagram recognition using 2d dynamic programming. Pattern Recognit 42:3215–3223

    Article  MATH  Google Scholar 

  13. Forsberg A, Dieterich M, Zeleznik R (1998) The music notepad. In: ACM symposium on user interface software and technology, San Francisco, CA, USA, pp 203–210

    Google Scholar 

  14. Fujiyama K, Iizuka K, Tanaka J (1999) Vic: Cmg input system using example figures. In: Proceedings of the international symposium on future software technology, Nanjing, China, pp 67–72

    Google Scholar 

  15. Gennari L, Kara LB, Stahovich TF, Shimada K (2005) Combining geometry and domain knowledge to interpret hand-drawn diagrams. Comput Graph 29(4):547–562

    Article  Google Scholar 

  16. Gross MD (1996) The electronic cocktail napkin—a computational environment for working with design diagrams. Des Stud 17:53–69

    Article  Google Scholar 

  17. Hall A, Pomm C, Widmayer P (2007) A combinatorial approach to multi-domain sketch recognition. In: EUROGRAPHICS workshop on sketch-based interfaces and modelling (SBIM’07)

    Google Scholar 

  18. Hammond T, Davis R (2005) Ladder, a sketching language for user interface developers. Comput Graph 29(4):518–532

    Article  Google Scholar 

  19. Jorge JA, Glinert EP (1995) Online parsing of visual languages using adjacency grammars. In: Proceedings of the eleventh international IEEE symposium on visual languages. IEEE Computer Society, Washington, pp 250–257

    Google Scholar 

  20. Kara LB, Stahovich TF (2004) Hierarchical parsing and recognition of hand-sketched diagrams. In: Proceedings of the 17th annual ACM symposium on user interface software and technology (UIST’04), pp 13–22

    Chapter  Google Scholar 

  21. Kim DH, Kim MJ (2006) A curvature estimation for pen input segmentation in sketch-based modeling. Comput Aided Des 36(3):238–248

    Article  Google Scholar 

  22. Kurtoglu T, Stahovich TF (2002) Interpreting schematic sketches using physical reasoning. In: Proceedings of the AAAI spring symposium on sketch understanding (SSSU’02), Palo Alto, CA, USA, pp 78–85

    Google Scholar 

  23. Landay JA, Myers BA (2001) Sketching interfaces: Toward more human interface design. Computer 34(3):56–64

    Article  Google Scholar 

  24. Lank E, Thorley JS, Chen S, Blostein D (2001) On-line recognition of uml diagrams. In: Proceedings of the sixth international conference on document analysis and recognition (ICDAR’01), Seattle, USA, pp 356–360

    Chapter  Google Scholar 

  25. Macé S, Anquetil E (2009) Eager interpretation of on-line hand-drawn structured documents: the Dali methodology. Pattern Recognit 42(12):3202–3214

    Article  MATH  Google Scholar 

  26. Macé S, Anquetil E, Bossis B (2007) Pen-based interaction for intuitive music composition and editing. In: Shen J, Shepherd J, Cui B, Liu L (eds) Intelligent music information systems: tools and methodologies, Idea Group, pp 261–288

  27. Mankoff J, Hudson SE, Abowd GD (2000) Oops: a toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces. Comput Graph 24(6):819–834

    Article  Google Scholar 

  28. Marriott K (1994) Constrain multiset grammars. In: Proceedings of the IEEE symposium on visual languages (VL’94), St Louis, MO, USA, pp 118–125

    Chapter  Google Scholar 

  29. Marriott K, Meyer B, Wittenburg K (1998) A survey of visual language specification and recognition. In: Marriott K, Meyer B (eds) Theory of visual languages. Springer, Berlin

    Chapter  Google Scholar 

  30. Mas J, Sánchez G, Lladós J, Lamiroy B (2007) An incremental on-line parsing algorithm for recognizing sketching diagrams. In: Proceedings of the ninth international conference on document analysis and recognition (ICDAR’07), Curitiba, Brazil, pp 452–456

    Chapter  Google Scholar 

  31. Mas J, Lladós J, Sánchez G, Jorge JA (2010) A syntactic approach based on distortion-tolerant adjacency grammars and a spatial-directed parser to interpret sketched diagrams. Pattern Recognit 43:4148–4164

    Article  MATH  Google Scholar 

  32. Miyao H, Maruyama M (2007) An online handwritten music symbol recognition system. Int J Doc Anal Recognit 9(1):49–58

    Article  Google Scholar 

  33. Mouchère H, Anquetil E (2006) A unified strategy to deal with different natures of reject. In: Proceedings of the international conference on pattern recognition (ICPR’06), Hong Kong, China, pp 792–795

    Google Scholar 

  34. Nakagawa M, Machii K, Kato N, Souya T (1993) Lazy recognition as a principle of pen interfaces. In: Proceedings of the conference companion on human factors in computing systems (INTERACT’93 and CHI’93), Amsterdam, The Netherlands, pp 89–90

    Chapter  Google Scholar 

  35. Plimmer B, Apperley M (2003) Software to sketch interface designs. In: Proceedings of the ninth international conference on human-computer interaction (INTERACT’03), Zürich, Switzerland, pp 73–80

    Google Scholar 

  36. Sezgin TM, Davis R (2007) Sketch interpretation using multiscale models of temporal patterns. IEEE Comput Graph Appl 27(1):28–37

    Article  Google Scholar 

  37. Sezgin TM, Stahovich T, Davis R (2001) Sketch based interfaces: Early processing for sketch understanding. In: Proceedings of international conference on computer graphics and interactive techniques (ICCGIT’01), San Diego, California, USA, pp 37–44

    Google Scholar 

  38. Shilman M, Pasula H, Russell S, Newton R (2002) Statistical visual language models for ink parsing. In: AAAI spring 2002 symposium on sketch understanding

    Google Scholar 

  39. Thimbleby W (2004) A novel pen-based calculator and its evaluation. In: Proceedings of the third Nordic conference on human-computer interaction, Tempere, Finland, pp 445–448

    Chapter  Google Scholar 

  40. Valois JP, Cote M, Cheriet M (2001) Online recognition of sketched electrical diagrams. In: Proceedings of the sixth international conference on document analysis and recognition (ICDAR’01), Seattle, USA, pp 460–464

    Chapter  Google Scholar 

  41. Zadeh LA (1994) Soft computing and fuzzy logic. IEEE Softw 11(6):48–56

    Article  Google Scholar 

  42. Zhao R (1993) Incremental recognition in gesture-based and syntax-directed diagram editors. In: Proceedings of the INTERACT ’93 and CHI ’93 conference on human factors in computing systems (CHI’93), Amsterdam, The Netherlands, pp 95–100

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. INSA–IRISA–UEB, Campus de Beaulieu, 35043, Rennes Cedex, France

    Sébastien Macé & Eric Anquetil

Authors
  1. Sébastien Macé
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Anquetil
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sébastien Macé.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Macé, S., Anquetil, E. Exploiting on-the-fly interpretation to design technical documents in a mobile context. J Multimodal User Interfaces 4, 129–145 (2011). https://doi.org/10.1007/s12193-011-0058-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-011-0058-4

Keywords

Search

Navigation