More Web Proxy on the site http://driver.im/

research-article

Chorus: a crowd-powered conversational assistant

Authors:

Walter S. Lasecki,

Jeffrey Nichols,

Anand Kulkarni,

James F. Allen,

Jeffrey P. BighamAuthors Info & Claims

UIST '13: Proceedings of the 26th annual ACM symposium on User interface software and technology

Pages 151 - 162

https://doi.org/10.1145/2501988.2502057

Published: 08 October 2013 Publication History

Abstract

Despite decades of research attempting to establish conversational interaction between humans and computers, the capabilities of automated conversational systems are still limited. In this paper, we introduce Chorus, a crowd-powered conversational assistant. When using Chorus, end users converse continuously with what appears to be a single conversational partner. Behind the scenes, Chorus leverages multiple crowd workers to propose and vote on responses. A shared memory space helps the dynamic crowd workforce maintain consistency, and a game-theoretic incentive mechanism helps to balance their efforts between proposing and voting. Studies with 12 end users and 100 crowd workers demonstrate that Chorus can provide accurate, topical responses, answering nearly 93% of user queries appropriately, and staying on-topic in over 95% of responses. We also observed that Chorus has advantages over pairing an end user with a single crowd worker and end users completing their own tasks in terms of speed, quality, and breadth of assistance. Chorus demonstrates a new future in which conversational assistants are made usable in the real world by combining human and machine intelligence, and may enable a useful new way of interacting with the crowds powering other systems.

References

[1]

Allen, J. F., Schubert, L. K., Ferguson, G., Heeman, P., Hwang, C. H., Kato, T., Light, M., Martin, N. G., Miller, B. W., Poesio, M., and Traum, D. R. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies. In Journal of Experimental and Theoretical AI (JETAI) 7 (1995), 7--48.

[2]

Bozzon, A., Brambilla, M., and Ceri, S Answering search queries with CrowdSearcher. In WWW 2012, 33--42.

Digital Library

[3]

Bernstein, M. S., Brandt, J. R., Miller, R. C., and Karger, D. R. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In UIST 2011, 33--42.

Digital Library

[4]

Bernstein, M. S., Karger, D. R., Miller, R. C., and Brandt, J. R. Analytic methods for optimizing realtime crowdsourcing. In Collective Intelligence 2012.

[5]

Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: a word processor with a crowd inside. In UIST 2010, 313--322.

Digital Library

[6]

Bernstein, M. S., Teevan, J., Dumais, S., Liebling, D., and Horvitz, E. Direct answers for search queries in the long tail. In CHI 2012, 237--246.

Digital Library

[7]

Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatarowicz, A., White, B., White, S., and Yeh, T. Vizwiz: nearly real-time answers to visual questions. In UIST 2010, 333--342.

Digital Library

[8]

Bilton, N. With apple's siri, a romance gone sour. http://bits.blogs.nytimes.com/2012/07/15/with-apple's-siri-a-romance-gone-sour/.

[9]

Bonnington, C. Steve jobs would have 'lost his mind' over Siri, former employee says. www.wired.com/gadgetlab/2012/05/apple-siri-disappointment/.

[10]

Chilton, L. Seaweed: A web application for designing economic games. Master's thesis, MIT, 2009.

[11]

Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popovic, Z., and Players, F. Predicting protein structures with a multiplayer online game. In Nature 466, 7307 (2010), 756--760.

[12]

Hirschman, L., Dahl, D. A., McKay, D. P., Norton, L. M., and Linebarger, M. C. Beyond class a: a proposal for automatic evaluation of discourse. In Workshop on Speech and Natural Language, HLT '90, ACL, 1990, 109--113.

Digital Library

[13]

Kittur, A., Smus, B., and Kraut, R. Crowdforge: Crowdsourcing complex work. In UIST 2011, 43--52.

Digital Library

[14]

Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M. and Horton, J. The future of crowd work. In CSCW 2013, 1301--1318.

Digital Library

[15]

Kulkarni, A. P., Can, M., and Hartmann, B. Turkomatic: automatic recursive task and workflow design for mechanical turk. In CHI 2011, 2053--2058.

Digital Library

[16]

Lasecki, W. S., Miller, C. D., Sadilek, A., AbuMoussa, A., R. Kushalnagar, and Bigham, J. Real-time captioning by groups of non-experts. In UIST 2012, 23--34.

Digital Library

[17]

Lasecki, W. S., Murray, K., White, S., Miller, R. C., and Bigham, J. P. Real-time crowd control of existing interfaces. In UIST 2011, 23--32.

Digital Library

[18]

Lasecki, W. S., Song, Y. C., Kautz, H., and Bigham, J. P. Real-time crowd labeling for deployable activity recognition. In CSCW 2013, 1203-1212.

Digital Library

[19]

Lasecki, W. S., Thiha, P., Zhong, Y., Brady, E. and Bigham, J. P. Answering Visual Questions with Conversational Crowd Assistants. In ASSETS 2013.

Digital Library

[20]

Lasecki, W. S., White, S., Murray, K. I., and Bigham, J. P. Crowd memory: Learning in the collective. In Collective Intelligence 2012.

[21]

Levitt, B., and March, J. G. Organizational learning. In Annual Review of Sociology 14 (1988), 319--340.

[22]

Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Turkit: human computation algorithms on Mechanical Turk. In UIST 2010, 57--66.

Digital Library

[23]

Sears, A., Lin, M., Jacko, J., Xiao, Y. When computers fade: Pervasive computing and situationally induced impairments and disabilities. In HCI International '03, 1298--1302.

[24]

Sun, Y.-A., Dance, C. R., Roy, S., and Little, G. How to assure the quality of human computation tasks when majority voting fails' In Workshop on Computational Social Science and the Wisdom of Crowds, NIPS 2011.

[25]

von Ahn, L. Human computation. In Ph.D. Thesis (2005).

Digital Library

[26]

von Ahn, L., and Dabbish, L. Labeling images with a computer game. In CHI 2004, 319--326.

Digital Library

[27]

Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A. Paradise: a framework for evaluating spoken dialogue agents. In Proceedings of the Association for Computational Linguistics, ACL '98, Association for Computational Linguistics (1997), 271--280.

Digital Library

[28]

Webb, N., Benyon, D., Hansen, P., and Mival, O. Evaluating human-machine conversation for appropriateness. In Proceedings of the Conference on Language Resources and Evaluation, N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, and D. Tapias, Eds., LREC'10, European Language Resources Association (ELRA) (May 2010).

[29]

Zhang, H., Law, E., Miller, R. C., Gajos, K. Z., Parkes, D. C., and Horvitz, E. Human computation tasks with global constraints. In CHI 2012, 217--226.

Digital Library

Cited By

Huang XZhang RLi YZhang BZhang JXu JXu S(2024)A Simulation and Training Platform for Remote-Sighted AssistanceSensors10.3390/s2423777324:23(7773)Online publication date: 4-Dec-2024
https://doi.org/10.3390/s24237773
Yu RLee SXie JBillah SCarroll J(2024)Human–AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM EraFuture Internet10.3390/fi1607025416:7(254)Online publication date: 18-Jul-2024
https://doi.org/10.3390/fi16070254
May LOhshiro KDang KSridhar SPai JFuentes MLee SCartwright M(2024)Unspoken Sound: Identifying Trends in Non-Speech Audio Captioning on YouTubeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642162(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642162
Show More Cited By

Index Terms

Chorus: a crowd-powered conversational assistant
1. Information systems
  1. Information systems applications

Recommendations

How many crowdsourced workers should a requester hire?

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Crowdsourcing a self-evolving dialog graph
CUI '19: Proceedings of the 1st International Conference on Conversational User Interfaces

In this paper we present a crowdsourcing-based approach for collecting dialog data for a social chat dialog system, which gradually builds a dialog graph from actual user responses and crowd-sourced system answers, conditioned by a given persona and ...
Towards Conversationally Intelligent Dialog Systems
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

Spoken dialog systems, lacking the means to address the complex phenomena of spontaneous speech and conversational dynamics, force users into a constrained mode of dialog that resembles text-based interaction more closely than spoken conversation. Turn-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '13: Proceedings of the 26th annual ACM symposium on User interface software and technology

October 2013

558 pages

ISBN:9781450322683

DOI:10.1145/2501988

General Chairs:
Shahram Izadi
Microsoft Research, UK
,
Aaron Quigley
University of St Andrews, UK
,
Program Chairs:
Ivan Poupyrev
Disney Research, USA
,
Takeo Igarashi
The University of Tokyo, Japan

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

UIST'13

Sponsor:

UIST'13: The 26th Annual ACM Symposium on User Interface Software and Technology

October 8 - 11, 2013

St. Andrews, Scotland, United Kingdom

Acceptance Rates

UIST '13 Paper Acceptance Rate 62 of 317 submissions, 20%;

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

96
Total Citations
View Citations
1,241
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)2

Reflects downloads up to 31 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang XZhang RLi YZhang BZhang JXu JXu S(2024)A Simulation and Training Platform for Remote-Sighted AssistanceSensors10.3390/s2423777324:23(7773)Online publication date: 4-Dec-2024
https://doi.org/10.3390/s24237773
Yu RLee SXie JBillah SCarroll J(2024)Human–AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM EraFuture Internet10.3390/fi1607025416:7(254)Online publication date: 18-Jul-2024
https://doi.org/10.3390/fi16070254
May LOhshiro KDang KSridhar SPai JFuentes MLee SCartwright M(2024)Unspoken Sound: Identifying Trends in Non-Speech Audio Captioning on YouTubeProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642162(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642162
Ehtesham-Ul-Haque MD’Rozario JAdnin RUtshaw FTasneem FShefa IAl Islam A(2024)EmoBot: Artificial emotion generation through an emotional chatbot during general-purpose conversationsCognitive Systems Research10.1016/j.cogsys.2023.10116883(101168)Online publication date: Jan-2024
https://doi.org/10.1016/j.cogsys.2023.101168
Chang CHe YDu XYang XXie H(2024)Dynamic Labeling: A Control System for Labeling Styles in Image Annotation TasksHuman Interface and the Management of Information10.1007/978-3-031-60107-1_8(99-118)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60107-1_8
Ma YAbbas TGadiraju U(2023)ContextBotProceedings of the 34th ACM Conference on Hypertext and Social Media10.1145/3603163.3609031(1-14)Online publication date: 4-Sep-2023
https://dl.acm.org/doi/10.1145/3603163.3609031
Xiao ZLiao QZhou MGrandison TLi Y(2023)Powering an AI Chatbot with Expert Sourcing to Support Credible Health Information AccessProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584031(2-18)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584031
Lei KYu MLewellen MKu VLee D(2023)Compass: Supporting Large Group Mentorship in a Chat-Based UIProceedings of the ACM on Human-Computer Interaction10.1145/35794707:CSCW1(1-25)Online publication date: 16-Apr-2023
https://dl.acm.org/doi/10.1145/3579470
Xie JYu RCui KLee SCarroll JBillah S(2023)Are Two Heads Better than One? Investigating Remote Sighted Assistance with Paired VolunteersProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596019(1810-1825)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596019
Abbas TGadiraju UKhan VMarkopoulos P(2022)Understanding User Perceptions of Response Delays in Crowd-Powered Conversational SystemsProceedings of the ACM on Human-Computer Interaction10.1145/35557656:CSCW2(1-42)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555765
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents