More Web Proxy on the site http://driver.im/

Article

Integrating intra and extra gestures into a mobile and multimodal shopping assistant

Authors:

Rainer Wasinger,

Antonio Krüger,

Oliver JacobsAuthors Info & Claims

PERVASIVE'05: Proceedings of the Third international conference on Pervasive Computing

Pages 297 - 314

https://doi.org/10.1007/11428572_18

Published: 08 May 2005 Publication History

Abstract

Accompanying the rise of mobile and pervasive computing, applications now need to adapt to their surrounding environments and provide users with information in the environment in an easy and natural manner. In this paper we describe a user interface that integrates multimodal input on a handheld device with external gestures performed with real world artifacts. The described approach extends reference resolution based on speech, handwriting and gesture to that of real world objects that users may hold in their hands. We discuss the varied interaction channels available to users that arise from mixing and matching input modalities on the mobile device with actions performed in the environment. We also discuss the underlying components required in handling these extended multimodal interactions and present an implementation of our ideas in a demonstrator called the Mobile ShopAssist. This demonstrator is then used as the basis for a recent usability study that we describe on user interaction within mobile contexts.

References

[1]

Baudel, T., Beaudouin-Lafon, M., CHARADE: Remote Control of Objects using Free-Hand Gestures, ACM Journal, Vol. 36, no. 7, 1993, pp. 28-35.

[2]

Bühler, D., Minker, W., Häußler, J., Krüger, S., Flexible Multimodal Human-Machine Interaction in Mobile Environments, In ICSLP, 2002, pp. 169-172.

[3]

Cohen, P., Johnston, M., McGee, M., Oviatt, S., Pittman, J., Simith, I., Chen, L., Clow, J., Quickset: Multimodal interaction for distributed applications, Proc. of ACM International Multimedia Conference, 1997, pp. 31-40.

[4]

Cole, R. A., Mariani, J., Uszkoreit, H., Zaenen A., Zue, V., Survey of the State of the Art in Human Language Technology, Book, Chp1, 1995.

[5]

Heckmann, D., "Introducing Situational Statements as an integrating Data Structure for User Modeling, Context-Awareness and Resource-Adaptive Computing", ABIS Workshop on adaptivity and user modelling in interactive software systems, 2003.

[6]

Kendon, A., Conducting Interaction: Patterns of behavior in focused encounters, Book: Cambridge University Press, 1990.

[7]

Kumar, S., Cohen, P. R., Coulston, R., Multimodal Interaction under Exerted Conditions in a Natural Field Setting. In Proc. of the Sixth International Conference on Multimodal Interfaces (ICMI), 2004, pp. 227-234.

[8]

Krüger, A., Butz, A., Müller, C., Stahl, C., Wasinger, R., Steinberg, K. E., Dirschl, A., The Connected User Interface: Realizing a Personal Situated Navigation Service, Proc. of the 9th International Conference on Intelligent User Interfaces, 2004, pp. 161-168.

[9]

Newcomb, E., Pashley, T., Stasko, J., Mobile Computing in the Retail Arena, Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI), 2003, pp. 337- 344.

[10]

Oviatt, S., DeAngeli, A., Kuhn, K., Integration and synchronization of input modes during multimodal human-computer interaction, In Proc. of CHI, 1997, pp. 415-422.

[11]

Oviatt, S. Mutual disambiguation of recognition errors in a multimodal architecture. In Proceedings of the Conference on Human Factors in Computing Systems, 1999, pp. 576- 583.

[12]

Pastel, R., Skalsky, N., Demonstrating Information in Simple Gestures, In Proc. of the 9th international conference on Intelligent User Interfaces, 2004, pp. 360-361.

[13]

Wahlster, W., Towards Symmetric Multimodality: Fusion and Fission of Speech, Gesture, and Facial Expression, In. Proc. of the 26th German Conference on Artificial Intelligence, 2003, pp. 1-18.

[14]

Wasinger, R., Krüger, A., Multi-modal Interaction with Mobile Navigation Systems, In: W. Wahlster (ed.): Special Journal Issue "Conversational User Interfaces", it - Information Technology 46 (2004) 6, München: Oldenbourg Wissenschaftsverlag (ISSN 1611-2776), 2004, pp. 322-331.

[15]

Weintraub, M., Taussig, K., Hunicke, K., and Snodgrass, A. Effect of speaking style on LVCSR performance. In Proc. of the International Conference on Spoken Language Processing, 1996, pp. 16-19.

Cited By

Wiehr FHirsch ASchmitz LKnieriemen NKrüger AKovtunova ABorgwardt SChang EDemberg VSteinmetz MHoffmann J(2021)Why Do I Have to Take Over Control? Evaluating Safe Handovers with Advance Notice and Explanations in HADProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479884(308-317)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479884
Meneweger TWilfinger DAslan IZachhuber DTscheligi MMattern FSantini SCanny JLangheinrich MRekimoto J(2013)Towards the counter free storeProceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication10.1145/2494091.2496009(999-1006)Online publication date: 8-Sep-2013
https://dl.acm.org/doi/10.1145/2494091.2496009
Dumas BSolórzano MSigner BRohs MSchmidt AAshbrook DRukzio E(2013)Design guidelines for adaptive multimodal mobile input solutionsProceedings of the 15th international conference on Human-computer interaction with mobile devices and services10.1145/2493190.2493227(285-294)Online publication date: 27-Aug-2013
https://dl.acm.org/doi/10.1145/2493190.2493227
Show More Cited By

Integrating intra and extra gestures into a mobile and multimodal shopping assistant
1. Hardware
  1. Power and energy
    1. Power estimation and optimization
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms

Recommendations

Mobile assistant: enhancing desktop interaction using mobile phone
ITS '12: Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces

Touchscreen of mobile phone is an important input channel, whereas most desktop computers have no such a modality. Furthermore, we observe a common situation where ubiquitous mobile phones are placed aside while users operate computers. In this poster, ...
Mobile shopping assistant: integration of mobile applications and web services
WWW '07: Proceedings of the 16th international conference on World Wide Web

The goal of this poster is to describe our implementation of a newarchitecture enabling efficient integration between mobile phoneapplications and Web Services. Using this architecture, we haveimplemented a mobile shopping assistant described further. ...
User-defined gestures for connecting mobile phones, public displays, and tabletops
MobileHCI '10: Proceedings of the 12th international conference on Human computer interaction with mobile devices and services

Gestures can offer an intuitive way to interact with a computer. In this paper, we investigate the question whether gesturing with a mobile phone can help to perform complex tasks involving two devices. We present results from a user study, where we ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

PERVASIVE'05: Proceedings of the Third international conference on Pervasive Computing

May 2005

346 pages

ISBN:3540260080

Editors:
Hans-W. Gellersen
Computing Department, Lancaster University, England
,
Roy Want
Computing Department, Intel Research, 2200 Mission College Blvd, Santa Clara, England
,
Albrecht Schmidt
Pervasive Computing and User Interface Engineering, University of Duisburg-Essen, 2200 Mission College Blvd, Santa Clara, Germany

Sponsors

INTEL: Intel Corporation
Microsoft Research: Microsoft Research
NTT DoCoMo

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 08 May 2005

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wiehr FHirsch ASchmitz LKnieriemen NKrüger AKovtunova ABorgwardt SChang EDemberg VSteinmetz MHoffmann J(2021)Why Do I Have to Take Over Control? Evaluating Safe Handovers with Advance Notice and Explanations in HADProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479884(308-317)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479884
Meneweger TWilfinger DAslan IZachhuber DTscheligi MMattern FSantini SCanny JLangheinrich MRekimoto J(2013)Towards the counter free storeProceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication10.1145/2494091.2496009(999-1006)Online publication date: 8-Sep-2013
https://dl.acm.org/doi/10.1145/2494091.2496009
Dumas BSolórzano MSigner BRohs MSchmidt AAshbrook DRukzio E(2013)Design guidelines for adaptive multimodal mobile input solutionsProceedings of the 15th international conference on Human-computer interaction with mobile devices and services10.1145/2493190.2493227(285-294)Online publication date: 27-Aug-2013
https://dl.acm.org/doi/10.1145/2493190.2493227
Gehring SLöchtefeld MDaiber FBöhmer MKrüger ADuarte CCarriço LJorge JOviatt SGonçalves D(2012)Using intelligent natural user interfaces to support sales conversationsProceedings of the 2012 ACM international conference on Intelligent User Interfaces10.1145/2166966.2166985(97-100)Online publication date: 14-Feb-2012
https://dl.acm.org/doi/10.1145/2166966.2166985
Aslan IMenon D(2009)Component-based development of mobile assistants with the ELEPHANT systemProceedings of the 6th International Conference on Mobile Technology, Application & Systems10.1145/1710035.1710044(1-8)Online publication date: 2-Sep-2009
https://dl.acm.org/doi/10.1145/1710035.1710044
Schmitz MQuraischy HVillar NIzadi SFraser MBenford S(2009)Tangible interaction with real and virtual productsProceedings of the 3rd International Conference on Tangible and Embedded Interaction10.1145/1517664.1517710(209-212)Online publication date: 16-Feb-2009
https://dl.acm.org/doi/10.1145/1517664.1517710
Schmitz MBaus JDörr R(2008)The digital sommelierProceedings of the 1st international conference on The internet of things10.5555/1793060.1793081(247-262)Online publication date: 26-Mar-2008
https://dl.acm.org/doi/10.5555/1793060.1793081
Uszkoreit HXu FLiu WSteffen JAslan ILiu JMüller CHoltkamp BWojciechowski M(2007)A successful field test of a mobile and multilingual information service system COMPASS2008Proceedings of the 12th international conference on Human-computer interaction: applications and services10.5555/1769451.1769571(1047-1056)Online publication date: 22-Jul-2007
https://dl.acm.org/doi/10.5555/1769451.1769571
Krüger ABaus JHeckmann DKruppa MWasinger R(2007)Adaptive mobile guidesThe adaptive web10.5555/1768197.1768217(521-549)Online publication date: 1-Jan-2007
https://dl.acm.org/doi/10.5555/1768197.1768217
Schmitz MKrüger ASchmidt SChin DZhou MLau TPuerta A(2007)Modelling personality in voices of talking products through prosodic parametersProceedings of the 12th international conference on Intelligent user interfaces10.1145/1216295.1216355(313-316)Online publication date: 28-Jan-2007
https://dl.acm.org/doi/10.1145/1216295.1216355
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents