More Web Proxy on the site http://driver.im/

article

Free access

Synergistic use of direct manipulation and natural language

Authors:

J. W. SullivanAuthors Info & Claims

ACM SIGCHI Bulletin, Volume 20, Issue SI

Pages 227 - 233

https://doi.org/10.1145/67450.67494

Published: 01 March 1989 Publication History

Abstract

This paper shows how the integration of natural language with direct manipulation produces a multimodal interface that overcomes limitations of these techniques when used separately. Natural language helps direct manipulation in being able to specify objects and actions by description, while direct manipulation enables users to learn which objects and actions are available in the system. Furthermore, graphical rendering and manipulation of context provides a partial solution to difficult problems of natural language anaphora.

References

[1]

S. Brennan, M. Friedman, and C. Pollard. A centering approach to pronouns. In Proceedings of the 25th Annual Meeting of the Association for Computational Linguistics, Stanford, California, 1987.

Digital Library

[2]

E. Charniak. Jack and Janet in search of a theory of knowledge. In Advance Papers of the Third Meeting of the International Joint Conference on Artificial Intelligence, William Kaufman, Inc., Los Altos, California, 1973.

[3]

K. Church and R. Patil. Coping with syntactic ambiguity or how to put the block in the box on the table. American Journal of Computational Linguistics, 8(3-4):139 - 149, 1982.

Digital Library

[4]

M. Dalrymple. The interpretation of tense and aspect in English. In Proceedings of the 26th Annual Meeting of the Association for Computational Linguistics, Buffalo, New York, 1988.

Digital Library

[5]

B. J. Grosz. Focusing and description in natural language dialogues, In Elements of Discourse Understanding, Cambridge University Press, 1981.

[6]

B.J. Grosz, A.K. Joshi, and S. Weinstein. Providing a unified account of definite noun phrases in discourse. In Proceedings of the 21st Annual Meeting of the Association for Computational Linguistics, Cambridge, Massachusetts, 1983.

Digital Library

[7]

P. J. Hayes. Using a knowledge base to drive an expert system interface with a natural language component. In J. A. Hendler, editor, Expert Systems: The User Interface, pages 153 - 182, Ablex, Norwood, NJ, 1988.

Digital Library

[8]

J. lZ. Hobbs and P. Martin. Local pragmatics. In Proceedings of the Ninth International joint Conference on Artificial Intelligence, Morgan Kaufman Publishers, Los Altos, California, August 1987.

[9]

J.R. Hobbs. Resolving pronoun reference. Lingua, 44, 1978.

Digital Library

[10]

R.A. Gargan Jr., J.W. Sullivan, and S.W. Tyler. Multimodal response planning: an adaptive rule based approach. In CH11988 Proceedings, May 1988.

Digital Library

[11]

J.G. Neal and S.C. Shapiro. intelligent multi-media interface technology. In Architectures for Intelligent interfaces: Elements and Prototypes, Monterey, CA, March 1988.

[12]

F.C.N. Pereira. Can drawing be liberated from the Von Neumann style? Proceedings of the 1983 A CM Database Week, A CM, 1983.

Digital Library

[13]

F.C.N. Pereira. Logic for natural language analysis. SRI Technical Note #275, Artificial Intelligence Center, SRI International, Menlo Park, Calif.

[14]

It. Reichman. Plain-speaking: A theory and grammar of spontaneous discourse. Ph.D. thesis, Department of Computer Science, Harvard University, 1981.

[15]

C.L. Sidner. Towards a computational theory of definite anaphora comprehension in English discourse. Technical Report 537, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1979.

Digital Library

[16]

H. R. Tennant, K. M. Ross, R. M. Saenz, C. W. Thompson, and J. R. Miller. Menubased natural language understanding. In Proceedings of the 21st Annual Meeting of the Association for Computational Linguistics, pages 151- 158, Cambridge, Massachusetts, June 1983.

Digital Library

[17]

S.W. Tyler, R.A. Gargan Jr., and J.W. Sullivan. Adaptation and response planning: elements of an intelligent interface architecture. In Third Annual Rocky Mountain Conference Proceedings, June 1988.

[18]

W. Wahlster. User and discourse models for multimodal communication. In J.W. Sullivan and S.W. Tyler, editors, Architectures for Intelligent Interfaces" Elements and Prototypes, Addison-Wesley, Palo Alto, CA, 1989.

[19]

D. Warren and F. Pereira. An efficient easily adaptable system for interpreting natural language queries. American Journal of Computational Linguistics, 8(3):110- 123, 1982.

Digital Library

Cited By

Pan LYu CHe ZShi Y(2023)A Human-Computer Collaborative Editing Tool for Conceptual DiagramsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580676(1-29)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580676
Peng JWang X(2023)User-Device-Interaction Model: A Multimodal Interaction Εvaluation System Based on Analytic Hierarchy ProcessHuman-Computer Interaction10.1007/978-3-031-35596-7_13(181-199)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35596-7_13
Faisandaz GGoguey AJouffrais CNigay L(2022)Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual ImpairmentProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556589(105-116)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556589
Show More Cited By

Index Terms

Synergistic use of direct manipulation and natural language
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Synergistic use of direct manipulation and natural language
CHI '89: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

This paper shows how the integration of natural language with direct manipulation produces a multimodal interface that overcomes limitations of these techniques when used separately. Natural language helps direct manipulation in being able to specify ...
Synergistic use of direct manipulation and natural language
Readings in intelligent user interfaces
Natural language and direct manipulation search tools in a multimodal information system
CHI EA '99: CHI '99 Extended Abstracts on Human Factors in Computing Systems

This study evaluated the usability of an existing, information-dense workspace, which places burdensome search requirements on its users. Direct manipulation principles were contrasted with natural language and multimodal interaction. The results ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGCHI Bulletin

ACM SIGCHI Bulletin Volume 20, Issue SI

March 1989

374 pages

ISSN:0736-6906

DOI:10.1145/67450

Editors:
K. Bice
Texas Instruments
,
C. Lewis
Univ. of Colorado, Boulder, CO

Issue’s Table of Contents

CHI '89: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
March 1989
397 pages
ISBN:0897913019
DOI:10.1145/67449
Editors:
K. Bice
Texas Instruments
,
C. Lewis
Univ. of Colorado, Boulder, CO

Copyright © 1989 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 1989

Published in SIGCHI Volume 20, Issue SI

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

100
Total Citations
View Citations
753
Total Downloads

Downloads (Last 12 months)238
Downloads (Last 6 weeks)18

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan LYu CHe ZShi Y(2023)A Human-Computer Collaborative Editing Tool for Conceptual DiagramsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580676(1-29)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580676
Peng JWang X(2023)User-Device-Interaction Model: A Multimodal Interaction Εvaluation System Based on Analytic Hierarchy ProcessHuman-Computer Interaction10.1007/978-3-031-35596-7_13(181-199)Online publication date: 9-Jul-2023
https://doi.org/10.1007/978-3-031-35596-7_13
Faisandaz GGoguey AJouffrais CNigay L(2022)Keep in Touch: Combining Touch Interaction with Thumb-to-Finger µGestures for People with Visual ImpairmentProceedings of the 2022 International Conference on Multimodal Interaction10.1145/3536221.3556589(105-116)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3536221.3556589
Ordones Raposo NCastro ACastro TLima DMakedon F(2022)Enhancing interaction of people with quadriplegiaProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529218(223-229)Online publication date: 29-Jun-2022
https://dl.acm.org/doi/10.1145/3529190.3529218
Giunchi DSztrajman AJames SSteed A(2021)Mixing Modalities of 3D Sketching and Speech for Interactive Model Retrieval in Virtual RealityProceedings of the 2021 ACM International Conference on Interactive Media Experiences10.1145/3452918.3458806(144-155)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3452918.3458806
Kim YLee BSrinivasan AChoe EKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch InteractionProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445421(1-17)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445421
Srinivasan ALee BStasko J(2021)Interweaving Multimodal Interaction With Flexible Unit Visualizations for Data ExplorationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.297805027:8(3519-3533)Online publication date: 1-Aug-2021
https://doi.org/10.1109/TVCG.2020.2978050
Rafael SAlmeida V(2021)Implementation Goals for Multimodal Interfaces in Human-Computer InteractionHuman-Computer Interaction. Theory, Methods and Tools10.1007/978-3-030-78462-1_17(230-239)Online publication date: 3-Jul-2021
https://doi.org/10.1007/978-3-030-78462-1_17
Saktheeswaran ASrinivasan AStasko J(2020)Touch? Speech? or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.297051226:6(2168-2179)Online publication date: 1-Jun-2020
https://doi.org/10.1109/TVCG.2020.2970512
Kim YDontcheva MAdar EHullman JBrewster SFitzpatrick GCox AKostakos V(2019)Vocal Shortcuts for Creative ExpertsProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300562(1-14)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300562
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents