Abstract
One frequent concern associated with the development of AI models is their perceived lack of transparency. Consequently, the AI academic community has been active in exploring mathematical approaches that can increase the explainability of models. However, ensuring explainability thoroughly in the real world remains an open question. Indeed, besides data scientists, a variety of users is involved in the model lifecycle with varying motivations and backgrounds. In this paper, we sought to better characterize these explanations needs. Specifically, we conducted a user research study within a large institution that routinely develops and deploys AI model. Our analysis led to the identification of five explanation focuses and three standard user profiles that together enable to better describe what explainability means in real life. We also propose a mapping between explanation focuses and a set of existing explainability approaches as a way to link the user view and AI-born techniques.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ribeiro, M.T., Singh, S., Guestrin, C.: Why Should I Trust You?: Explaining the Predictions of Any Classifier (2016)
Lundberg, S.M., Lee, S.: A Unified Approach to Interpreting Model Predictions
Kim, B., Austin, U.T.: Examples are not enough, learn to criticize! criticism for interpretability. In: NIPS (2016)
Wang, D., Yang, Q., Abdul, A., Lim, B.Y.: Designing theory-driven user-centric explainable AI (2019). https://doi.org/10.1145/3290605.3300831
Liao, Q.V., Gruen, D., Miller, S.: Questioning the AI: informing design practices for explainable AI user experiences, pp. 1–15 (2020). https://doi.org/10.1145/3313831.3376590
Gunning, D.: Explainable artificial intelligence (XAI). In: Defense Advanced Research Projects Agency (DARPA) (2017)
Lim, B.Y., Dey, A.K.: Assessing demand for intelligibility in context-aware applications. In: ACM's International Conference Proceedings Series, pp. 195–204 (2009). https://doi.org/10.1145/1620545.1620576
F. Doshi-Velez and B. Kim, “A Roadmap for a Rigorous Science of Interpretability,” no. Ml, pp. 1–13, 2017.
Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2018). https://doi.org/10.1016/j.artint.2018.07.007
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Pedreschi, D., Giannotti, F.:A survey of methods for explaining black box models, arXiv, pp. 1–45 (2018)
Chari, S., Seneviratne, O., Gruen, D.M., Foreman, M.A., Das, A.K., McGuinness, D.L.: Explanation ontology: a model of explanations for user-centered AI. In: Pan, J.Z., et al. (eds.) ISWC. LNCS, vol. 12507, pp. 228–243. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62466-8_15
Barredo Arrieta, A., et al.: Explainable explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012
Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: An overview of interpretability of machine learning. In: Proceedings of 2018 IEEE 5th International Conference on Data Science and Advanced Analytics, DSAA 2018, pp. 80–89 (2019). https://doi.org/10.1109/DSAA.2018.00018
Arya, V., et al.: One explanation does not fit all: a toolkit and taxonomy of AI explainability techniques (2019). http://arxiv.org/abs/1909.03012.
Parekh, J., Mozharovskyi, P., d’Alche-Buc, F.: A framework to learn with interpretation (2020)
Sokol, K., Flach, P.: Explainability fact sheets: a framework for systematic assessment of explainable approaches. In: FAT*2020 - Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 56–67 (2020). https://doi.org/10.1145/3351095.3372870
Belle, V., Papantonis, I.: Principles and practice of explainable machine learning. arXiv (2020)
Molnar, C., Casalicchio, G., Bischl, B.: Interpretable machine learning – a brief history, state-of-the-art and challenges. In: Koprinska, I., et al. (eds.) ECML. CCIS, vol. 1323, pp. 417–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65965-3_28
Vilone, G., Longo, L.: Explainable artificial intelligence: a systematic review. arXiv, no. Dl (2020)
Freitas, A.A.: Comprehensible classification models. ACM SIGKDD Explor. Newsl. 15(1), 1 (2014). https://doi.org/10.1145/2594473.2594475
Fisher, A., Rudin, C., Dominici, F.: All models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously. arXiv (2018)
Apley, D.W., Zhu, J.: Visualizing the effects of predictor variables in black box supervised learning models. J. R. Stat. Soc. Ser. B Stat. Methodol. 82(4), 1059–1086 (2020). https://doi.org/10.1111/rssb.12377
Zhao, Q., Hastie, T.: Causal interpretations of black-box models. Department of Statistics, Stanford University (2016)
Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 1527–1535 (2018)
Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E.: Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J. Comput. Graph. Stat. 24(1), 44–65 (2015). https://doi.org/10.1080/10618600.2014.907095
Gurumoorthy, K.S., Dhurandhar, A., Cecchi, G., Aggarwal, C.: Efficient data representation by selecting prototypes with importance weights. In: Proceedings of IEEE International Conference on Data Mining, ICDM, vol. 2019, pp. 260–269 (2019). https://doi.org/10.1109/ICDM.2019.00036
Wachter, S., Mittelstadt, B., Russell, C.: Counterfactual explanations without opening the black box: automated decisions and the GDPR. SSRN Electron. J. 31, 1–52 (2017). https://doi.org/10.2139/ssrn.3063289
Dandl, S., Molnar, C., Binder, M., Bischl, B.: Multi-objective counterfactual explanations. In: Bäck, T., et al. (eds.) PPSN. LNCS, vol. 12269, pp. 448–469. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58112-1_31
Samek, W., Wiegand, T., Müller, K.R.: Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. arXiv (2017)
Nunes, I., Jannach, D.: A systematic review and taxonomy of explanations in decision support and recommender systems
Hilton, D.J., Slugoski, B.R.: Knowledge-based causal attribution. The abnormal conditions focus model. Psychol. Rev. 93(1), 75–88 (1986). https://doi.org/10.1037/0033-295X.93.1.75
Lim, B.Y., Dey, A.K.: Investigating intelligibility for uncertain context-aware applications. In: UbiComp 2011, Proceedings of the 2011 ACM Conference on Ubiquitous Computing, pp. 415–424 (2011). https://doi.org/10.1145/2030112.2030168
Lim, B.Y., Dey, A.K.: Evaluating intelligibility usage and usefulness in a context-aware application. In: Kurosu, M. (ed.) HCI. LNCS, vol. 8008, pp. 92–101. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39342-6_11
Krause, J., Perer, A., Ng, K.: Interacting with predictions: visual inspection of black-box machine learning models. In: Conference on Human Factors Computing Systems - Proceedings, pp. 5686–5697 (2016). https://doi.org/10.1145/2858036.2858529
Coppers, S., et al.: Intellingo: an intelligible translation environment. In: Conference on Human Factors Computing System - Proc., vol. 2018-April, (2018). https://doi.org/10.1145/3173574.3174098
Lim, B.Y., Dey, A.K.: Toolkit to support intelligibility in context-aware applications, p. 13 (2010). https://doi.org/10.1145/1864349.1864353
Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., Hussmann, H.: Bringing transparency design into practice, pp. 211–223 (2018). https://doi.org/10.1145/3172944.3172961
Kulesza, C., Principles, S.: Principles of explanatory debugging to personalize interactive machine learning (2015). https://doi.org/10.1145/2678025.2701399
Maria, R.: How to analyze qualitative data from UX research : thematic analysis. Nielsen Norman Group Publication (2019)
Ribera, M., Lapedriza, A.: Can we do better explanations? A proposal of user-centered explainable AI. In: CEUR Workshop Proceedings, vol. 2327 (2019)
Lim, B.Y., Dey, A.K.: Evaluating intelligibility usage and usefulness in a context-aware application
Lim, B.Y., Dey, A.K.: Investigating intelligibility for uncertain context-aware applications (2011)
Dhurandhar, A., et al.: Explanations based on the missing: towards contrastive explanations with pertinent negatives. arXiv, NeurIPS (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, L., Wang, H., Deleris, L.A. (2021). What Does It Mean to Explain? A User-Centered Study on AI Explainability. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2021. Lecture Notes in Computer Science(), vol 12797. Springer, Cham. https://doi.org/10.1007/978-3-030-77772-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-77772-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77771-5
Online ISBN: 978-3-030-77772-2
eBook Packages: Computer ScienceComputer Science (R0)