More Web Proxy on the site http://driver.im/

research-article

Marcelle: Composing Interactive Machine Learning Workflows and Interfaces

Authors:

Jules Françoise,

Baptiste Caramiaux,

Téo SanchezAuthors Info & Claims

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Pages 39 - 53

https://doi.org/10.1145/3472749.3474734

Published: 12 October 2021 Publication History

Abstract

Human-centered approaches to machine learning have established theoretical foundations, design principles and interaction techniques to facilitate end-user interaction with machine learning systems. Yet, general-purpose toolkits supporting the design of interactive machine learning systems are still missing, despite their potential to foster reuse, appropriation and collaboration between different stakeholders including developers, machine learning experts, designers and end users. In this paper, we present an architectural model for toolkits dedicated to the design of human interactions with machine learning. The architecture is built upon a modular collection of interactive components that can be composed to build interactive machine learning workflows, using reactive pipelines and composable user interfaces. We introduce Marcelle, a toolkit for the design of human interactions with machine learning that implements this model. We illustrate Marcelle with two implemented case studies: (1) a HCI researcher conducts user studies to understand novice interaction with machine learning, and (2) a machine learning expert and a clinician collaborate to develop a skin cancer diagnosis system. Finally, we discuss our experience with the toolkit, along with its limitation and perspectives.

References

[1]

[n.d.]. Feathers | A framework for real-time applications and REST APIs. https://feathersjs.com/. (Accessed on 04/06/2021).

[2]

[n.d.]. GitHub - fossasia/visdom: A flexible tool for creating, organizing, and sharing visualizations of live, rich data. Supports Torch and Numpy.https://github.com/fossasia/visdom. (Accessed on 04/06/2021).

[3]

[n.d.]. Keras: The Python Deep Learning API. https://keras.io/. (Accessed on 07/27/2021).

[4]

[n.d.]. Lobe | Machine Learning Made Easy. https://lobe.ai/. (Accessed on 04/06/2021).

[5]

[n.d.]. ml5js: Friendly Machine Learning For The Web. https://ml5js.org/. (Accessed on 04/06/2021).

[6]

[n.d.]. ml.js: Machine learning tools in JavaScript. https://github.com/mljs/ml. (Accessed on 04/06/2021).

[7]

[n.d.]. Most.js: Monadic Event Stream. https://github.com/mostjs/core/. (Accessed on 04/07/2021).

[8]

[n.d.]. Neptune.ai | Metadata Store for MLOps. https://neptune.ai/. (Accessed on 04/06/2021).

[9]

[n.d.]. Runway | Make the Impossible. https://runwayml.com/. (Accessed on 04/06/2021).

[10]

[n.d.]. Svelte: Cybernetically enhanced web apps. https://svelte.dev/. (Accessed on 04/07/2021).

[11]

[n.d.]. VisualDL: Deep Learning Visualization Toolkit. https://github.com/PaddlePaddle/VisualDL. (Accessed on 04/06/2021).

[12]

[n.d.]. Vite: Next Generation Frontend Tooling. https://vitejs.dev/. (Accessed on 04/06/2021).

[13]

Cycling ’74. [n.d.]. Max is software for experimentation and invention. https://cycling74.com/. (Accessed on 04/06/2021).

[14]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Savannah, GA, 265–283. https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi

Digital Library

[15]

Allegro AI. [n.d.]. ClearML | MLOps for Data Science Teams. https://clear.ml/. (Accessed on 04/06/2021).

[16]

Saleema Amershi. 2011. Designing for Effective End-User Interaction with Machine Learning. In Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology(Santa Barbara, California, USA) (UIST ’11 Adjunct). Association for Computing Machinery, New York, NY, USA, 47–50. https://doi.org/10.1145/2046396.2046416

Digital Library

[17]

Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (12 2014), 105–120. https://doi.org/10.1609/aimag.v35i4.2513

Digital Library

[18]

Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. ModelTracker: Redesigning Performance Analysis Tools for Machine Learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 337–346. https://doi.org/10.1145/2702123.2702509

Digital Library

[19]

Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233

Digital Library

[20]

Geering Up at UBC. 2019. Bringing AI into the Classroom. https://www.actua.ca/en/bringing-ai-into-the-classroom. [Online; accessed 16-Sep-2020].

[21]

Engineer Bainomugisha, Andoni Lombide Carreton, Tom van Cutsem, Stijn Mostinckx, and Wolfgang de Meuter. 2013. A Survey on Reactive Programming. Comput. Surveys 45, 4, Article 52 (Aug. 2013), 34 pages. https://doi.org/10.1145/2501654.2501666

[22]

Michel Beaudouin-Lafon. 2017. Towards Unified Principles of Interaction. In Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (Cagliari, Italy) (CHItaly ’17). Association for Computing Machinery, New York, NY, USA, Article 1, 2 pages. https://doi.org/10.1145/3125571.3125602

Digital Library

[23]

Stuart Berg, Dominik Kutra, Thorben Kroeger, Christoph N. Straehle, Bernhard X. Kausler, Carsten Haubold, Martin Schiegg, Janez Ales, Thorsten Beier, Markus Rudy, Kemal Eren, Jaime I. Cervantes, Buote Xu, Fynn Beuttenmueller, Adrian Wolny, Chong Zhang, Ullrich Koethe, Fred A. Hamprecht, and Anna Kreshuk. 2019. Ilastik: Interactive Machine Learning for (Bio)Image Analysis. Nature Methods 16, 12 (Dec. 2019), 1226–1232. https://doi.org/10.1038/s41592-019-0582-9

[24]

Francisco Bernardo, Michael Zbyszyński, Mick Grierson, and Rebecca Fiebrink. 2020. Designing and Evaluating the Usability of a Machine Learning API for Rapid Prototyping Music Technology. Frontiers in Artificial Intelligence 3 (2020), 13. https://doi.org/10.3389/frai.2020.00013

[25]

Jamie Bullock and Ali Momeni. 2015. Ml.Lib: Robust, Cross-Platform, Open-Source Machine Learning for Max and Pure Data. In Proceedings of the International Conference on New Interfaces for Musical Expression(Baton Rouge, Louisiana, USA) (NIME 2015). The School of Music and the Center for Computation and Technology (CCT), Louisiana State University, Baton Rouge, Louisiana, USA, 265–270.

Digital Library

[26]

Baptiste Caramiaux, Jules Françoise, Norbert Schnell, and Frédéric Bevilacqua. 2014. Mapping through listening. Computer Music Journal 38, 3 (2014), 34–48.

Digital Library

[27]

Michelle Carney, Barron Webster, Irene Alvarado, Kyle Phillips, Noura Howell, Jordan Griffith, Jonas Jongejan, Amit Pitaru, and Alexander Chen. 2020. Teachable Machine: Approachable Web-Based Tool for Exploring Machine Learning Classification. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3334480.3382839

Digital Library

[28]

Eric Corbett, Nathaniel Saul, and Meg Pirrung. [n.d.]. Interactive Machine Learning Heuristics. Technical Report. https://learningfromusersworkshop.github.io/papers/IMLH.pdf

[29]

Janez Demšar, Tomaž Curk, Aleš Erjavec, Črt Gorup, Tomaž Hočevar, Mitar Milutinovič, Martin Možina, Matija Polajnar, Marko Toplak, Anže Starič, Miha Štajdohar, Lan Umek, Lan Žagar, Jure Žbontar, Marinka Žitnik, and Blaž Zupan. 2013. Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14, 35 (2013), 2349–2353. http://jmlr.org/papers/v14/demsar13a.html

Digital Library

[30]

Carlos Gonzalez Diaz, Phoenix Perry, and Rebecca Fiebrink. 2019. Interactive Machine Learning for More Expressive Game Interactions. In 2019 IEEE Conference on Games (CoG). 1–2. https://doi.org/10.1109/CIG.2019.8848007

[31]

Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. Association for Computing Machinery, New York, NY, USA, 278–288. https://doi.org/10.1145/3025453.3025739

Digital Library

[32]

John J. Dudley and Per Ola Kristensson. 2018. A Review of User Interface Design for Interactive Machine Learning. ACM Transactions on Interactive Intelligent Systems 8, 2, Article 8 (June 2018), 37 pages. https://doi.org/10.1145/3185517

Digital Library

[33]

Jerry Alan Fails and Dan R. Olsen. 2003. Interactive Machine Learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (Miami, Florida, USA) (IUI ’03). Association for Computing Machinery, New York, NY, USA, 39–45. https://doi.org/10.1145/604045.604056

Digital Library

[34]

Rebecca Fiebrink and Baptiste Caramiaux. 2018. The Machine Learning Algorithm as Creative Musical Tool. In The Oxford Handbook of Algorithmic Music, Roger T. Dean and Alex McLean (Eds.). Oxford University Press, 181–208. https://research.gold.ac.uk/id/eprint/23154/

[35]

Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems - CHI ’11. ACM Press, Vancouver, BC, Canada, 147. https://doi.org/10.1145/1978942.1978965

Digital Library

[36]

James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive Concept Learning in Image Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 29–38. https://doi.org/10.1145/1357054.1357061

Digital Library

[37]

Jules Françoise and Frédéric Bevilacqua. 2018. Motion-Sound Mapping through Interaction: An Approach to User-Centered Design of Auditory Feedback Using Machine Learning. ACM Transactions on Interactive Intelligent Systems 8, 2, Article 16 (June 2018), 30 pages. https://doi.org/10.1145/3211826

Digital Library

[38]

Jules Françoise, Norbert Schnell, Riccardo Borghesi, and Frédéric Bevilacqua. 2014. Probabilistic Models for Designing Motion and Sound Relationships. In Proceedings of the International Conference on New Interfaces for Musical Expression. Goldsmiths, University of London, London, United Kingdom, 287–292. https://doi.org/10.5281/zenodo.1178764

[39]

Marco Gillies. 2019. Understanding the Role of Interactive Machine Learning in Movement Interaction Design. ACM Transactions on Computer-Human Interaction 26, 1, Article 5 (Feb. 2019), 34 pages. https://doi.org/10.1145/3287307

Digital Library

[40]

Marco Gillies, Rebecca Fiebrink, Atau Tanaka, Jérémie Garcia, Frédéric Bevilacqua, Alexis Heloir, Fabrizio Nunnari, Wendy Mackay, Saleema Amershi, Bongshin Lee, Nicolas d’Alessandro, Joëlle Tilmanne, Todd Kulesza, and Baptiste Caramiaux. 2016. Human-Centred Machine Learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 3558–3565. https://doi.org/10.1145/2851581.2856492

Digital Library

[41]

Markus Hofmann and Ralf Klinkenberg. 2013. RapidMiner: Data Mining Use Cases and Business Analytics Applications. Chapman & Hall/CRC.

Digital Library

[42]

Fred Hohman, Kanit Wongsuphasawat, Mary Beth Kery, and Kayur Patel. 2020. Understanding and Visualizing Data Iteration in Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376177

Digital Library

[43]

Andreas Holzinger. 2016. Interactive machine learning for health informatics: when do we need the human-in-the-loop?Brain Informatics 3, 2 (6 2016), 119–131. https://doi.org/10.1007/s40708-016-0042-6

[44]

Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arxiv:1704.04861

[45]

M. Kahng, N. Thorat, D. H. Chau, F. B. Viégas, and M. Wattenberg. 2019. GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation. IEEE Transactions on Visualization and Computer Graphics 25, 1(2019), 310–320. https://doi.org/10.1109/TVCG.2018.2864500

Digital Library

[46]

Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter development team. 2016. Jupyter Notebooks ? a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas, Fernando Loizides and Birgit Scmidt (Eds.). IOS Press, 87–90. https://eprints.soton.ac.uk/403913/

[47]

Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (Atlanta, Georgia, USA) (IUI ’15). Association for Computing Machinery, New York, NY, USA, 126–137. https://doi.org/10.1145/2678025.2701399

Digital Library

[48]

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. UMAP: Uniform Manifold Approximation and Projection. Journal of Open Source Software 3, 29 (2018), 861. https://doi.org/10.21105/joss.00861

[49]

Microsoft. [n.d.]. ONNX.js: run ONNX models using JavaScript. https://github.com/Microsoft/onnxjs/. (Accessed on 04/06/2021).

[50]

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). Association for Computing Machinery, New York, NY, USA, 220–229. https://doi.org/10.1145/3287560.3287596

Digital Library

[51]

Giselle Nodalo, Jose Ma. Santiago, Jolene Valenzuela, and Jordan Aiko Deja. 2019. On Building Design Guidelines for An Interactive Machine Learning Sandbox Application. In Proceedings of the 5th International ACM In-Cooperation HCI and UX Conference (Jakarta, Surabaya, Bali, Indonesia) (CHIuXiD’19). Association for Computing Machinery, New York, NY, USA, 70–77. https://doi.org/10.1145/3328243.3328253

Digital Library

[52]

Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, USA.

Digital Library

[53]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf

Digital Library

[54]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825–2830.

[55]

Wojciech Samek, Gregoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Muller. 2019. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (1st ed.). Springer Publishing Company, Incorporated.

Digital Library

[56]

Téo Sanchez, Baptiste Caramiaux, Jules Françoise, Frédéric Bevilacqua, and Wendy E. Mackay. 2021. How Do People Train a Machine? Strategies and (Mis)Understandings. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1, Article 162 (April 2021), 26 pages. https://doi.org/10.1145/3449236

[57]

Hugo Scurto, Bavo Van Kerrebroeck, Baptiste Caramiaux, and Frédéric Bevilacqua. 2021. Designing Deep Reinforcement Learning for Human Parameter Exploration. ACM Transactions on Computer-Human Interaction 28, 1, Article 1 (Jan. 2021), 35 pages. https://doi.org/10.1145/3414472

Digital Library

[58]

Shital Shah, Roland Fernandez, and Steven Drucker. 2019. A System for Real-Time Interactive Analysis of Deep Learning Training. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Valencia, Spain) (EICS ’19). Association for Computing Machinery, New York, NY, USA, Article 16, 6 pages. https://doi.org/10.1145/3319499.3328231

Digital Library

[59]

Daniel Smilkov, Shan Carter, D Sculley, Fernanda B Viégas, and Martin Wattenberg. 2016. Direct-Manipulation Visualization of Deep Networks. In KDD 2016 Workshop on Interactive Data Exploration and Analytics (IDEA). San Francisco, California, USA. arxiv:1708.03788

[60]

Daniel Smilkov, Nikhil Thorat, Yannick Assogba, Ann Yuan, Nick Kreeger, Ping Yu, Kangyi Zhang, Shanqing Cai, Eric Nielsen, David Soergel, Stan Bileschi, Michael Terry, Charles Nicholson, Sandeep N. Gupta, Sarah Sirajuddin, D. Sculley, Rajat Monga, Greg Corrado, Fernanda B. Viegas, and Martin Wattenberg. 2019. TensorFlow.js: Machine Learning for the Web and Beyond. (2019). https://arxiv.org/abs/1901.05350

[61]

Thilo Spinner, Udo Schlegel, Hanna Schäfer, and Mennatallah El-Assady. 2020. explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning. IEEE Transactions on Visualization and Computer Graphics 26, 1(2020), 1064–1074. https://doi.org/10.1109/TVCG.2019.2934629

[62]

Justin Talbot, Bongshin Lee, Ashish Kapoor, and Desney S. Tan. 2009. EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). Association for Computing Machinery, New York, NY, USA, 1283–1292. https://doi.org/10.1145/1518701.1518895

Digital Library

[63]

Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, John Paoli, Susana Puig, Cliff Rosendahl, H. Peter Soyer, Iris Zalaudek, and Harald Kittler. 2020. Human–Computer Collaboration for Skin Cancer Recognition. Nature Medicine 26, 8 (Aug. 2020), 1229–1234. https://doi.org/10.1038/s41591-020-0942-0

[64]

Emily Wall, Soroush Ghorashi, and Gonzalo Ramos. 2019. Using Expert Patterns in Assisted Interactive Machine Learning: A Study in Machine Teaching. In Human-Computer Interaction – INTERACT 2019, David Lamas, Fernando Loizides, Lennart Nacke, Helen Petrie, Marco Winckler, and Panayiotis Zaphiris (Eds.). Springer International Publishing, Cham, 578–599.

Digital Library

[65]

Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y. Lim. 2019. Designing Theory-Driven User-Centric Explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300831

Digital Library

[66]

Z. J. Wang, R. Turko, O. Shaikh, H. Park, N. Das, F. Hohman, M. Kahng, and D. H. Polo Chau. 2021. CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization. IEEE Transactions on Visualization and Computer Graphics 27, 2(2021), 1396–1406. https://doi.org/10.1109/TVCG.2020.3030418

[67]

Kanit Wongsuphasawat, Daniel Smilkov, James Wexler, Jimbo Wilson, Dandelion Mané, Doug Fritz, Dilip Krishnan, Fernanda B. Viégas, and Martin Wattenberg. 2018. Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow. IEEE Transactions on Visualization and Computer Graphics 24, 1(2018), 1–12. https://doi.org/10.1109/TVCG.2017.2744878

[68]

Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-Examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376301

Digital Library

[69]

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How Transferable Are Features in Deep Neural Networks?. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (Montreal, Canada) (NIPS’14). MIT Press, Cambridge, MA, USA, 3320–3328.

Digital Library

[70]

Jun Yuan, Changjian Chen, Weikai Yang, Mengchen Liu, Jiazhi Xia, and Shixia Liu. 2021. A Survey of Visual Analytics Techniques for Machine Learning. Computational Visual Media 7, 1 (March 2021), 3–36. https://doi.org/10.1007/s41095-020-0191-7

Cited By

Hernandez JColom M(2025)Reproducible research policies and software/data management in scientific computing journals: a survey, discussion, and perspectivesFrontiers in Computer Science10.3389/fcomp.2024.14918236Online publication date: 15-Jan-2025
https://doi.org/10.3389/fcomp.2024.1491823
Muralikumar MMcDonald D(2025)An Emerging Design Space of How Tools Support Collaborations in AI Design and DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/37011819:1(1-28)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701181
Tsiakas KMurray-Rust D(2024)Unpacking Human-AI interactions: From Interaction Primitives to a Design SpaceACM Transactions on Interactive Intelligent Systems10.1145/366452214:3(1-51)Online publication date: 8-Jun-2024
https://dl.acm.org/doi/10.1145/3664522
Show More Cited By

Recommendations

A Review of User Interface Design for Interactive Machine Learning
Special Issue on Human-Centered Machine Learning

Interactive Machine Learning (IML) seeks to complement human perception and intelligence by tightly integrating these strengths with the computational power and speed of computers. The interactive process is designed to involve input from the user but ...
Emerging Perspectives in Human-Centered Machine Learning
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

Current Machine Learning (ML) models can make predictions that are as good as or better than those made by people. The rapid adoption of this technology puts it at the forefront of systems that impact the lives of many, yet the consequences of this ...
Human-in-the-loop machine learning: a state of the art
Abstract
Researchers are defining new types of interactions between humans and machine learning algorithms generically called human-in-the-loop machine learning. Depending on who is in control of the learning process, we can identify: active learning, in ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 2021

1357 pages

ISBN:9781450386357

DOI:10.1145/3472749

Editors:
Jeffrey Nichols
Apple, USA
,
Ranjitha Kumar
UIUC, USA
,
Michael Nebeling
University of Michigan, USA

Copyright © 2021 ACM.

© 2021 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Agence Nationale de la Recherche

Conference

UIST '21

Sponsor:

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 10 - 14, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
1,025
Total Downloads

Downloads (Last 12 months)199
Downloads (Last 6 weeks)14

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hernandez JColom M(2025)Reproducible research policies and software/data management in scientific computing journals: a survey, discussion, and perspectivesFrontiers in Computer Science10.3389/fcomp.2024.14918236Online publication date: 15-Jan-2025
https://doi.org/10.3389/fcomp.2024.1491823
Muralikumar MMcDonald D(2025)An Emerging Design Space of How Tools Support Collaborations in AI Design and DevelopmentProceedings of the ACM on Human-Computer Interaction10.1145/37011819:1(1-28)Online publication date: 10-Jan-2025
https://dl.acm.org/doi/10.1145/3701181
Tsiakas KMurray-Rust D(2024)Unpacking Human-AI interactions: From Interaction Primitives to a Design SpaceACM Transactions on Interactive Intelligent Systems10.1145/366452214:3(1-51)Online publication date: 8-Jun-2024
https://dl.acm.org/doi/10.1145/3664522
Lee YLam MVasconcelos HBernstein MFinn C(2024)Clarify: Improving Model Robustness With Natural Language CorrectionsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676362(1-19)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676362
Das Antar AMolaei SChen YLee MBanovic N(2024)VIME: Visual Interactive Model Explorer for Identifying Capabilities and Limitations of Machine Learning Models for Sequential Decision-MakingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676323(1-21)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676323
Kim DShin HYadgarova SSon JSubramonyam HKim J(2024)AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and ClientsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661577(728-742)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661577
Lee CLee MMutlu B(2024)The AI-DEC: A Card-based Design Method for User-centered AI ExplanationsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661576(1010-1028)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661576
Szymanski MOoge JDe Croon RVanden Abeele VVerbert K(2024)Feedback, Control, or Explanations? Supporting Teachers With Steerable Distractor-Generating AIProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636933(690-700)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636933
Scully-Allison CLumsden IWilliams KBartels JTaufer MBrink SBhatele APearce OIsaacs K(2024)Design Concerns for Integrated Scripting and Interactive Visualization in Notebook EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.335456130:9(6572-6585)Online publication date: Sep-2024
https://doi.org/10.1109/TVCG.2024.3354561
Sungeelee VLoriette ASigaud OCaramiaux B(2024)Interactive curriculum learning increases and homogenizes motor smoothnessScientific Reports10.1038/s41598-024-53253-314:1Online publication date: 3-Feb-2024
https://doi.org/10.1038/s41598-024-53253-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents