Abstract
Natural language interface (NLI) enables an efficient and effective interaction by allowing a user to submit a single phrase in natural language to the system. Free hand gestures can be added to an NLI to specify the referents for deictic terms in speech. By combining NLI with other modalities to a multimodal user interface, speech utterance length can be reduced, and users need not clearly specify the referent verbally. Integrating deictic terms with deictic gestures is a critical function in multimodal user interface. This paper presents a novel approach to extend chart parsing used in natural language processing (NLP) to integrate multimodal input based on speech and manual deictic gesture. The effectiveness of the technique has been validated through experiments, using a traffic incident management scenario where an operator interacts with a map on large display at a distance and issues multimodal commands through speech and manual gestures. The preliminary experiment of the proposed algorithm shows encouraging results.
Chapter PDF
Similar content being viewed by others
References
Bird, S., Klein, E., Loper, E.: Parsing (2005), In http://nltk.sourceforge.net
Chen, F., Choi, E., Epps, J., Lichman, S., Ruiz, N., Shi, Y., Taib, R., Wu, M.A.: Study of Manual Gesture-Based Selection for the PEMMI Multimodal Transport Management Interface. In: Proceedings of ICMI’05, October 4–6, Trento, Italy, pp. 274–281 (2005)
Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and Evaluation of a Constraint-Based Multimodal Fusion System for Speech and 3D Pointing Gestures. In: Proceedings of ICMI 2004, October 13-15, State College Pennsylvania, USA, pp. 175–182 (2004)
Johnston, M.: Unification-based Multimodal Parsing. In: Proceedings of ACL’1998, Montreal, Quebec, Canada, pp. 624–630. ACM, New York (1998)
Johnston, M., Bangalore, S.: Finite-state multimodal parsing and understanding. In: Proceedings of COLING 2000, Saarbrücken, Germany, pp. 369–375 (2000)
Kaiser, E., Demirdjian, D., Gruenstein, A., Li, X., Niekrasz, J., Wesson, M., Kumar, S., Demo.: A Multimodal Learning Interface for Sketch, Speak and Point Creation of a Schedule Chart. In: Proceedings of ICMI’04, October 13-15, State College Pennsylvania, USA, pp. 329-330 (2004)
Latoschik, M.E.: A User Interface Framework for Multimodal VR Interactions. In: Proc. ICMI 2005 (2005)
Sun, Y., Chen, F., Shi, Y., Chung, V.: A Novel Method for Multi-sensory Data Fusion in Multimodal Human Computer Interaction. In: Proc. OZCHI 2006 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sun, Y., Chen, F., Shi, Y., Chung, V. (2007). An Input-Parsing Algorithm Supporting Integration of Deictic Gesture in Natural Language Interface. In: Jacko, J.A. (eds) Human-Computer Interaction. HCI Intelligent Multimodal Interaction Environments. HCI 2007. Lecture Notes in Computer Science, vol 4552. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73110-8_22
Download citation
DOI: https://doi.org/10.1007/978-3-540-73110-8_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-73108-5
Online ISBN: 978-3-540-73110-8
eBook Packages: Computer ScienceComputer Science (R0)