More Web Proxy on the site http://driver.im/

research-article

Open access

Eliciting Spoken Interruptions to Inform Proactive Speech Agent Design

Authors:

Justin Edwards,

Christian Janssen,

Benjamin R. CowanAuthors Info & Claims

CUI '21: Proceedings of the 3rd Conference on Conversational User Interfaces

Article No.: 23, Pages 1 - 12

https://doi.org/10.1145/3469595.3469618

Published: 27 July 2021 Publication History

All formats PDF

Abstract

Current speech agent interactions are typically user-initiated, limiting the interactions they can deliver. Future functionality will require agents to be proactive, sometimes interrupting users. Little is known about how these spoken interruptions should be designed, especially in urgent interruption contexts. We look to inform design of proactive agent interruptions through investigating how people interrupt others engaged in complex tasks. We therefore developed a new technique to elicit human spoken interruptions of people engaged in other tasks. We found that people interrupted sooner when interruptions were urgent. Some participants used access rituals to forewarn interruptions, but most rarely used them. People balanced speed and accuracy in timing interruptions, often using cues from the task they interrupted. People also varied phrasing and delivery of interruptions to reflect urgency. We discuss how our findings can inform speech agent design and how our paradigm can help gain insight into human interruptions in new contexts.

Supplementary Material

a23-edwards-supplement (a23-edwards-supplement.pdf)

Tables

Download
48.08 KB

References

[1]

R. H. Baayen, D. J. Davidson, and D. M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59, 4 (Nov. 2008), 390–412. https://doi.org/10.1016/j.jml.2007.12.005

[2]

Brian P. Bailey and Shamsi T. Iqbal. 2008. Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Transactions on Computer-Human Interaction 14, 4 (Jan. 2008), 1–28. https://doi.org/10.1145/1314683.1314689

Digital Library

[3]

Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68, 3 (April 2013), 255–278. https://doi.org/10.1016/j.jml.2012.11.001

[4]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1–48. https://doi.org/10.18637/jss.v067.i01

[5]

Jelmer P. Borst, Niels A. Taatgen, and Hedderik van Rijn. 2010. The problem state: A cognitive bottleneck in multitasking.Journal of Experimental Psychology: Learning, Memory, and Cognition 36, 2(2010), 363–382. https://doi.org/10.1037/a0018106

[6]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (Dec. 2006). https://doi.org/10.1191/1478088706qp063oa Publisher: Taylor & Francis (Routledge).

[7]

Duncan P. Brumby, Anna L. Cox, Jonathan Back, and Sandy J. J. Gould. 2013. Recovering from an interruption: Investigating speed-accuracy trade-offs in task resumption behavior.Journal of Experimental Psychology: Applied 19, 2 (2013), 95–107. https://doi.org/10.1037/a0032696

[8]

Duncan P. Brumby, Samantha C.E. Davies, Christian P. Janssen, and Justin J. Grace. 2011. Fast or safe?: how performance objectives determine modality output choices while interacting on the move. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI ’11. ACM Press, Vancouver, BC, Canada, 473. https://doi.org/10.1145/1978942.1979009

Digital Library

[9]

Narae Cha, Auk Kim, Cheul Young Park, Soowon Kang, Minkyu Park, Jae-Gil Lee, Sangsu Lee, and Uichin Lee. 2019. “Hello There! Is Now a Good Time to Talk?’’: Opportune Moments for Proactive Interactions with Smart Speakers. 4, 3 (2019), 28.

[10]

Leigh Clark, Cosmin Munteanu, Vincent Wade, Benjamin R. Cowan, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, and Christine Murad. 2019. What Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300705

Digital Library

[11]

Laura Dabbish, Gloria Mark, and Víctor M González. 2011. Why do i keep interrupting myself?: environment, habit and self-interruption. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3127–3130.

Digital Library

[12]

Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, and Benjamin R. Cowan. 2019. Mapping Perceptions of Humanness in Intelligent Personal Assistant Interaction. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services - MobileHCI ’19. ACM Press, Taipei, Taiwan, 1–12. https://doi.org/10.1145/3338286.3340116

Digital Library

[13]

Jens Edlund, Julia Bell Hirschberg, and Mattias Heldner. 2009. Pause and gap length in face-to-face interaction. Columbia University (2009). https://doi.org/10.7916/d82f7wt9

[14]

Justin Edwards, He Liu, Zhou Tianyu, Gould Gould, Sandy J. J., Leigh Clark, Philip Doyle, and Benjamin R Cowan. 2019. Multitasking with Alexa: How Using Intelligent Personal Assistants Impacts Language-based Primary Task Performance. In Proceedings of the 1st International Conference on Conversational User Interfaces. Dublin, Ireland. Accetped.

Digital Library

[15]

Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods 5, 1 (March 2006), 80–92. https://doi.org/10.1177/160940690600500107 Publisher: SAGE Publications Inc.

[16]

Emer Gilmartin, Marine Collery, Ketong Su, Yuyun Huang, Christy Elias, Benjamin R. Cowan, and Nick Campbell. 2017. Social talk: making conversation with people and machine. In Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents - ISIAA 2017. ACM Press, Glasgow, UK, 31–32. https://doi.org/10.1145/3139491.3139494

Digital Library

[17]

Erving Goffman. 1971. Relations in public: microstudies of the public order. Basic Books, New York, NY, USA. OCLC: 699515377.

[18]

Wayne D. Gray and Deborah A. Boehm-Davis. 2000. Milliseconds matter: An introduction to microstrategies and to their use in describing and predicting interactive behavior.Journal of Experimental Psychology: Applied 6, 4 (2000), 322–335. https://doi.org/10.1037/1076-898X.6.4.322

[19]

Elizabeth Hellier, Judy Edworthy, Ben Weedon, Kathryn Walters, and Austin Adams. 2002. The Perceived Urgency of Speech Warnings: Semantics versus Acoustics. Human Factors: The Journal of the Human Factors and Ergonomics Society 44, 1 (March 2002), 1–17. https://doi.org/10.1518/0018720024494810

[20]

William J. Horrey and Mary F. Lesch. 2009. Driver-initiated distractions: Examining strategic adaptation for in-vehicle task initiation. Accident Analysis & Prevention 41, 1 (Jan. 2009), 115–122. https://doi.org/10.1016/j.aap.2008.10.008

[21]

Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI ’99. ACM Press, Pittsburgh, Pennsylvania, United States, 159–166. https://doi.org/10.1145/302979.303030

Digital Library

[22]

Shamsi T. Iqbal and Brian P. Bailey. 2005. Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In CHI ’05 extended abstracts on Human factors in computing systems - CHI ’05. ACM Press, Portland, OR, USA, 1489. https://doi.org/10.1145/1056808.1056948

[23]

Christian P. Janssen, Duncan P. Brumby, and Rae Garnett. 2012. Natural Break Points: The Influence of Priorities and Cognitive and Motor Cues on Dual-Task Interleaving. Journal of Cognitive Engineering and Decision Making 6, 1 (March 2012), 5–29. https://doi.org/10.1177/1555343411432339

[24]

Christian P. Janssen, Sandy J.J. Gould, Simon Y.W. Li, Duncan P. Brumby, and Anna L. Cox. 2015. Integrating knowledge of multitasking and interruptions across different perspectives and research methods. International Journal of Human-Computer Studies 79 (July 2015), 1–5. https://doi.org/10.1016/j.ijhcs.2015.03.002

Digital Library

[25]

David Kieras, David Meyer, James Ballas, and Erick Lauber. 2000. Modern Computational Perspectives on Executive Mental Processes and Cognitive Control: Where to from Here?

[26]

Paul D. Krivonos and Mark L. Knapp. 1975. Initiating communication: What do you say when you say hello?Central States Speech Journal 26, 2 (June 1975), 115–125. https://doi.org/10.1080/10510977509367829

[27]

Tate T. Kubose, Kathryn Bock, Gary S. Dell, Susan M. Garnsey, Arthur F. Kramer, and Jeff Mayhugh. 2006. The effects of speech production and speech comprehension on simulated driving performance. Applied Cognitive Psychology 20, 1 (Jan. 2006), 43–63. https://doi.org/10.1002/acp.1164

[28]

Jakob Landesberger, Ute Ehrlich, and Wolfgang Minker. 2020. Do the Urgent Things first! - Detecting Urgency in Spoken Utterances based on Acoustic Features. In Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization. ACM, Genoa Italy, 53–58. https://doi.org/10.1145/3386392.3397598

Digital Library

[29]

Jakob Landesberger, Ute Ehrlich, and Wolfgang Minker. 2020. ”What is it?” How to Collect Urgent Utterances using a Gamification Approach. In 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, Virtual Event DC USA, 19–22. https://doi.org/10.1145/3409251.3411713

[30]

John K. Lindstedt and Wayne D. Gray. 2019. Distinguishing experts from novices by the Mind’s Hand and Mind’s Eye. Cognitive Psychology 109 (March 2019), 1–25. https://doi.org/10.1016/j.cogpsych.2018.11.003

[31]

Ewa Luger and Abigail Sellen. 2016. ”Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. ACM Press, Santa Clara, California, USA, 5286–5297. https://doi.org/10.1145/2858036.2858288

Digital Library

[32]

Nikolas Martelaro, Jaime Teevan, and Shamsi T. Iqbal. 2019. An Exploration of Speech-Based Productivity Support in the Car. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300494

Digital Library

[33]

Daniel C. McFarlane. 2002. Comparison of Four Primary Methods for Coordinating the Interruption of People in Human-Computer Interaction. Human–Computer Interaction 17, 1 (March 2002), 63–139. https://doi.org/10.1207/S15327051HCI1701_2

[34]

Lotte Meteyard and Robert A.I. Davies. 2020. Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language 112 (June 2020), 104092. https://doi.org/10.1016/j.jml.2020.104092

[35]

R Core Team. 2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

[36]

Dario D. Salvucci. 2005. A Multitasking General Executive for Compound Continuous Tasks. Cognitive Science 29, 3 (2005), 457–492. https://doi.org/10.1207/s15516709cog0000_19

[37]

Dario D. Salvucci and Niels A. Taatgen. 2008. Threaded cognition: An integrated theory of concurrent multitasking.Psychological Review 115, 1 (2008), 101–130. https://doi.org/10.1037/0033-295X.115.1.101

[38]

Rob Semmens, Nikolas Martelaro, Pushyami Kaveti, Simon Stent, and Wendy Ju. 2019. Is Now A Good Time?: An Empirical Study of Vehicle-Driver Communication Timing. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/3290605.3300867

Digital Library

[39]

Sarah M. Simmons, Jeff K. Caird, and Piers Steel. 2017. A meta-analysis of in-vehicle and nomadic voice-recognition system interaction and driving performance. Accident Analysis & Prevention 106 (Sept. 2017), 31–43. https://doi.org/10.1016/j.aap.2017.05.013

[40]

Remo M.A. van der Heiden, Shamsi T. Iqbal, and Christian P. Janssen. 2017. Priming Drivers before Handover in Semi-Autonomous Cars. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17. ACM Press, Denver, Colorado, USA, 392–404. https://doi.org/10.1145/3025453.3025507

Digital Library

[41]

Martijn H. Vastenburg, David V. Keyson, and Huib de Ridder. 2008. Considerate home notification systems: a field study of acceptability of notifications in the home. Personal and Ubiquitous Computing 12, 8 (Nov. 2008), 555–566. https://doi.org/10.1007/s00779-007-0176-x

Digital Library

[42]

Priscilla N. Y. Wong, Duncan P. Brumby, Harsha Vardhan Ramesh Babu, and Kota Kobayashi. 2019. Voices in Self-Driving Cars Should be Assertive to More Quickly Grab a Distracted Driver’s Attention. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications(AutomotiveUI ’19). Association for Computing Machinery, Utrecht, Netherlands, 165–176. https://doi.org/10.1145/3342197.3344535

Digital Library

[43]

Yunhan Wu, Justin Edwards, Orla Cooney, Anna Bleakley, Philip R. Doyle, Leigh Clark, Daniel Rough, and Benjamin R. Cowan. 2020. Mental Workload and Language Production in Non-Native Speaker IPA Interaction. In Proceedings of the 2nd Conference on Conversational User Interfaces. ACM, Bilbao Spain, 1–8. https://doi.org/10.1145/3405755.3406118

Digital Library

[44]

Yunhan Wu, Daniel Rough, Anna Bleakley, Justin Edwards, Orla Cooney, Philip R. Doyle, Leigh Clark, and Benjamin R. Cowan. 2020. See What I’m Saying? Comparing Intelligent Personal Assistant Use for Native and Non-Native Language Speakers. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, Oldenburg Germany, 1–9. https://doi.org/10.1145/3379503.3403563

Digital Library

Cited By

Edwards JNguyen ASobocinski MLämsä Jde Araujo ADang BWhitehead RRoberts AKaarlela MJarvela S(2024)MAI - A Proactive Speech Agent for Metacognitive Mediation in Collaborative LearningProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665585(1-5)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665585
Rey BSakamoto YSin JIrani P(2024)Understanding User Preferences of Voice Assistant Answer Structures for Personal Health Data QueriesProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665552(1-15)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665552
Edwards JDoyle PBranigan HCowan B(2024)Comparing Perceptions of Static and Adaptive Proactive Speech AgentsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665548(1-12)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665548
Show More Cited By

Recommendations

Comparing Perceptions of Static and Adaptive Proactive Speech Agents
CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces

A growing literature on speech interruptions describes how people interrupt one another with speech, but these behaviours have not yet been implemented in the design of artificial agents which interrupt. Perceptions of a prototype proactive speech agent ...
A diary study of task switching and interruptions
CHI '04: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We report on a diary study of the activities of information workers aimed at characterizing how people interleave multiple tasks amidst interruptions. The week-long study revealed the type and complexity of activities performed, the nature of the ...
Not Merely Deemed as Distraction: Investigating Smartphone Users’ Motivations for Notification-Interaction
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Notifications are commonly considered a distraction when they arrive during a task, and consequently, prior research has consistently sought effective ways of deferring their arrival until task transitions. However, many smartphone users still interact ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

CUI '21: Proceedings of the 3rd Conference on Conversational User Interfaces

July 2021

262 pages

ISBN:9781450389983

DOI:10.1145/3469595

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

ADAPT - Centre for Digital Content Technology

Conference

CUI '21

CUI '21: CUI 2021 - 3rd Conference on Conversational User Interfaces

July 27 - 29, 2021

Bilbao (online), Spain

Acceptance Rates

Overall Acceptance Rate 34 of 100 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,086
Total Downloads

Downloads (Last 12 months)342
Downloads (Last 6 weeks)109

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Edwards JNguyen ASobocinski MLämsä Jde Araujo ADang BWhitehead RRoberts AKaarlela MJarvela S(2024)MAI - A Proactive Speech Agent for Metacognitive Mediation in Collaborative LearningProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665585(1-5)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665585
Rey BSakamoto YSin JIrani P(2024)Understanding User Preferences of Voice Assistant Answer Structures for Personal Health Data QueriesProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665552(1-15)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665552
Edwards JDoyle PBranigan HCowan B(2024)Comparing Perceptions of Static and Adaptive Proactive Speech AgentsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665548(1-12)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665548
Ma YZiegler JBoratto LGena CMarras MGermanakos PPopescus E(2024)The Effect of Proactive Cues on the Use of Decision Aids in Conversational Recommender SystemsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665186(305-315)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3665186
Bahnsen KTiemann LPlabst LGrundgeiger T(2024)Augmented Reality Cues Facilitate Task Resumption after Interruptions in Computer-Based and Physical TasksProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642666(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642666
Jaber RZhong SKuoppamäki SHosseini AGessinger IBrumby DCowan BMcmillan D(2024)Cooking With Agents: Designing Context-aware Voice InteractionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642183(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642183
Lingler ATalypova DJokinen JOulasvirta AWintersberger P(2024)Supporting Task Switching with Reinforcement LearningProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642063(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642063
Ma YZiegler J(2024)Investigating meta-intents: user interaction preferences in conversational recommender systemsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09411-3Online publication date: 24-Sep-2024
https://doi.org/10.1007/s11257-024-09411-3
Zargham NReicherts LAvanesi VRogers YMalaka R(2023)Tickling Proactivity: Exploring the Use of Humor in Proactive Voice AssistantsProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627777(294-320)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627777
Dubiel MBongard-Blanchy KLeiva LSergeeva A(2023)Are you sure you want to order that?Proceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3604312(1-6)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3571884.3604312
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents