More Web Proxy on the site http://driver.im/

abstract

Wizard of Oz Prototyping for Machine Learning Experiences

Author:

Jacob T. BrowneAuthors Info & Claims

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: LBW2621, Pages 1 - 6

https://doi.org/10.1145/3290607.3312877

Published: 02 May 2019 Publication History

Abstract

Machine learning is being adopted in a wide range of products and services. Despite its adoption, design and research processes for machine learning experiences have yet to be cemented in the user experience community. Prototyping machine learning experiences is noted to be particularly challenging. This paper suggests Wizard of Oz prototyping to help designers incorporate human-centered design processes into the development of machine learning experiences. This paper also surfaces a set of topics to consider in evaluating Wizard of Oz machine learning prototypes.

References

[1]

Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine 35, 4 (2014), 105--120. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2513

Digital Library

[2]

Saleema Amershi. 2012. Designing for Effective End-User Interaction with Machine Learning. Ph.D. Dissertation. University of Washington, Seattle, WA.

Digital Library

[3]

Tone Bratteteig and Guri Verne. 2018. Does AI make PD obsolete?: exploring challenges from artificial intelligence to participatory design. In Proceedings of the 15th Participatory Design Conference: Short Papers, Situated Actions, Workshops and Tutorial - Volume 2 (PDC '18), Liesbeth Huybrechts, Maurizio Teli, Ann Light, Yanki Lee, Carl Di Salvo, Erik Grönvall, Anne Marie Kanstrup, and Keld Bødker (Eds.), Vol. 2. ACM, New York, NY, USA, Article 8, 5 pages.

Digital Library

[4]

Ajay Chander, Ramya Srinivasan, Suhas Chelian, Jun Wang, and Kanji Uchino. Working with beliefs: AI transparency in the enterprise. In Joint Proceedings of the ACM IUI 2018 Workshops co-located with the 23rd ACM Conference on Intelligent User Interfaces (ACM IUI 2018), 2018.

[5]

Ajay Chander and Ramya Srinivasan. 2018. Evaluating Explanations by Cognitive Value. In: Holzinger A., Kieseberg P., Tjoa A., Weippl E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2018. Lecture Notes in Computer Science, vol 11015. Springer, Cham

[6]

Eric Corbett, Nathaniel Saul, and Meg Pirrung. Interactive Machine Learning Heuristics. (October 2018). Retrieved December 1, 2018 from https://learningfromusersworkshop.github.io/papers/IMLH.pdf

[7]

Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces (IUI '93), Wayne D. Gray, William E. Hefley, and Dianne Murray (Eds.). ACM, New York, NY, USA, 193--200.

Digital Library

[8]

Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, Philipp Ebel. 2019. The Future of Human-AI Collaboration: A Taxonomy of Design Knowledge for Hybrid Intelligence Systems. 2019. - Hawaii International Conference on System Sciences (HICSS). - Maui, Hawaii, USA.

[9]

Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 278--288.

Digital Library

[10]

Steven Dow, Blair MacIntyre, Jaemin Lee, Christopher Oezbek, Jay David Bolter, and Maribeth Gandy. 2005. Wizard of Oz Support throughout an Iterative Design Process. IEEE Pervasive Computing 4, 4 (October 2005), 18--26.

Digital Library

[11]

John J. Dudley and Per Ola Kristensson. 2018. A Review of User Interface Design for Interactive Machine Learning. ACM Trans. Interact. Intell. Syst. 1, 1, Article 1 (March 2018), 37 pages.

Digital Library

[12]

Marco Gillies, Rebecca Fiebrink, Atau Tanaka, Jérémie Garcia, Frédéric Bevilacqua, Alexis Heloir, Fabrizio Nunnari, Wendy Mackay, Saleema Amershi, Bongshin Lee, Nicolas d'Alessandro, Joëlle Tilmanne, Todd Kulesza, and Baptiste Caramiaux. 2016. Human-Centred Machine Learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 3558--3565.

Digital Library

[13]

John D. Gould, John Conti, and Todd Hovanyecz. 1983. Composing letters with a simulated listening typewriter. Commun. ACM 26, 4 (April 1983), 295--308.

Digital Library

[14]

Bruce Hanington and Bella Martin. 2012. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Rockport Publishers, Beverly, MA.

[15]

Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E. Imel, and David C. Atkins. 2017. Designing Contestability: Interaction Design, Machine Learning, and Mental Health. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, New York, NY, USA, 95--99.

Digital Library

[16]

Tom Hitron, Iddo Wald, Hadas Erel, and Oren Zuckerman. 2018. Introducing children to machine learning concepts through hands-on experience. In Proceedings of the 17th ACM Conference on Interaction Design and Children (IDC '18). ACM, New York, NY, USA, 563--568.

Digital Library

[17]

Kristina Höök. 2000. Steps to take before intelligent user interfaces become real. Interacting with Computers 12, 4 (2000), 409--426.

[18]

J. F. Kelley. 1983. An empirical methodology for writing user-friendly natural language computer applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '83), Ann Janda (Ed.). ACM, New York, NY, USA, 193--196.

Digital Library

[19]

Michael van Lent, William Fisher, and Michael Mancuso. 2004. An Explainable Artificial Intelligence System for Smallunit Tactical Behavior.

[20]

Josh Lovejoy and Jess Holbrook. 2017. Human-Centered Machine Learning. (July 2017). Retrieved December 3, 2018 from https://medium.com/google-design/human-centered-machine-learning-a770d10562cd

[21]

David Maulsby, Saul Greenberg, and Richard Mander. 1993. Prototyping an intelligent agent through Wizard of Oz. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, New York, NY, USA, 277--284.

Digital Library

[22]

Kathryn McElroy. 2017. Prototyping for Designers. O'Reilly Media, Sebastopol, CA.

[23]

Jakob Nielsen and Rolf Molich. 1990. Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '90), Jane Carrasco Chew and John Whiteside (Eds.). ACM, New York, NY, USA, 249--256.

Digital Library

[24]

Christopher Noessel. 2017. Designing Agentive Technology. Rosenfield Media, Brooklyn, NY.

Digital Library

[25]

Don Norman. 2009. The Design of Future Things. Basic Books, New York, NY.

[26]

Masayuki Okamoto, Yeonsoo Yang and Toru Ishida. 2001. Wizard of Oz Method for Learning Dialog Agents, Proceedings of the 5th International Workshop on Cooperative Information Agents V, p.20--25, September 06-08, 2001

Digital Library

[27]

Raquel Torres Peralta, Tasneem Kaochar, Ian R. Fasel, Clayton T. Morrison, Thomas J. Walsh, and Paul R. Cohen. 2011. Challenges to decoding the intention behind natural instruction. 2011 Ro-Man (2011).

[28]

Edward Tiong, Olivia Seow, Bradley A Camburn, Kenneth Teo, Arlindo Silva, Kristin Lee Wood, Dan Jensen, and Maria Yeng. 2018. The Economies and Dimensionality of Design Prototyping: Value, Time, Cost and Fidelity (DETC201885747). ASME. J. Mech. Des. 2018;():.

[29]

Jasper van der Waa, Jurriaan van Diggelen, and Mark Neerincx. 2018. The design and validation of an intuitive confidence measure. In Workshop On Explainable Smart Systems (EXSS), 2018.

[30]

Todd Zaki Warfel. 2009. Protoyping: A Practitioner's Guide. Rosenfeld Media, Brooklyn, NY.

[31]

Jason Williams and Steve Young. 2003. Using wizard-of-oz simulations to bootstrap reinforcement-learning-based dialog management systems. In Proc. 4th SIGdial workshop.

[32]

Qian Yang, Jina Suh, Nan-Chen Chen, and Gonzalo Ramos. 2018. Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 573--584.

Digital Library

[33]

Qian Yang, John Zimmerman, Aaron Steinfeld, and Anthony Tomasic. 2016. Planning Adaptive Mobile Experiences When Wireframing. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16). ACM, New York, NY, USA, 565--576.

Digital Library

[34]

Qian Yang. 2018. Machine learning as a UX design material: How can we imagine beyond automation, recommenders, and reminders? In AAAI Spring Symposium Series

Cited By

Garofalo MFantini APellugrini RPilato GVillari MGiannotti F(2025)Conversational XAI: Formalizing Its Basic Design PrinciplesMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74627-7_22(295-309)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74627-7_22
Baudoux GSafin S(2024)Study of computer multi-instrumented reflexive conversation activity in preliminary architectural designInternational Journal of Architectural Computing10.1177/14780771241310207Online publication date: 20-Dec-2024
https://doi.org/10.1177/14780771241310207
Baudoux G(2024)The benefits and challenges of artificial intelligence image generators for architectural ideation: Study of an alternative human-machine co-creation exchange based on sketch recognitionInternational Journal of Architectural Computing10.1177/14780771241253438Online publication date: 13-May-2024
https://doi.org/10.1177/14780771241253438
Show More Cited By

Index Terms

Wizard of Oz Prototyping for Machine Learning Experiences
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods

Recommendations

Prototyping an intelligent agent through Wizard of Oz
CHI '93: Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems

Turvy is a simulated prototype of an instructible agent. The user teaches it by demonstrating actions and pointing at or talking about relevant data. We formalized our assumptions about what could be implemented, then used the Wizard of Oz to flesh out ...
Wizard of Oz experiments for companions
BCS-HCI '09: Proceedings of the 23rd British HCI Group Annual Conference on People and Computers: Celebrating People and Technology

Wizard of Oz experiments allow designers and developers to see the reactions of people as they interact with to-be-developed technologies. At the Centre for Interaction Design at Edinburgh Napier University we are developing a Wizard of Oz system to ...
Wizard of Oz experiments and companion dialogues
BCS '10: Proceedings of the 24th BCS Interaction Specialist Group Conference

Novel speech systems such as the conversational agents being developed by the Companions Project (www.companions-project.org) can be simulated using the Wizard of Oz methodology. In this approach technologies that are not yet ready for testing by people ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

3673 pages

ISBN:9781450359719

DOI:10.1145/3290607

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Check for updates

Author Tags

Qualifiers

Abstract

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
1,269
Total Downloads

Downloads (Last 12 months)104
Downloads (Last 6 weeks)13

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Garofalo MFantini APellugrini RPilato GVillari MGiannotti F(2025)Conversational XAI: Formalizing Its Basic Design PrinciplesMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74627-7_22(295-309)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74627-7_22
Baudoux GSafin S(2024)Study of computer multi-instrumented reflexive conversation activity in preliminary architectural designInternational Journal of Architectural Computing10.1177/14780771241310207Online publication date: 20-Dec-2024
https://doi.org/10.1177/14780771241310207
Baudoux G(2024)The benefits and challenges of artificial intelligence image generators for architectural ideation: Study of an alternative human-machine co-creation exchange based on sketch recognitionInternational Journal of Architectural Computing10.1177/14780771241253438Online publication date: 13-May-2024
https://doi.org/10.1177/14780771241253438
Seneviratne AGrowns BLukosch S(2024)On the Impact of a Simulated Cognitive Augmentation to Detect Deception on Decision-making ConfidenceProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688901(1-2)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688901
Schäfer AFreyer M(2024)"Is There Anything Else...?" Role-Play Exploration To Use Response Times As A Relationship-Building Effect With Voice AssistantsProceedings of Mensch und Computer 202410.1145/3670653.3677500(474-478)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1145/3670653.3677500
Fang JArechiga NNamikoshi KBravo NHogan CShamma D(2024)On LLM Wizards: Identifying Large Language Models' Behaviors for Wizard of Oz ExperimentsProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673967(1-11)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673967
Overney CSaldías BDimitrakopoulou DRoy D(2024)SenseMate: An Accessible and Beginner-Friendly Human-AI Platform for Qualitative Data AnalysisProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645194(922-939)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645194
Deshpande MPark JPait SMagerko B(2024)Perceptions of Interaction Dynamics in Co-Creative AI: A Comparative Study of Interaction Modalities in DrawctoProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656202(102-116)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656202
Longdon JWesterlaken MBlackwell AGabrys JOssom BAshton-Butt AAcheampong E(2024)Justice-oriented Design Listening: Participatory Ecoacoustics with a Ghanaian Forest CommunityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3643044(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3643044
Rezk ASimkute ALuger EVines JElsden CEvans MJones R(2024)Agency Aspirations: Understanding Users' Preferences And Perceptions Of Their Role In Personalised News CurationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642634(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642634
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten