research-article

Open access

The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.

Author:

Zachary C. LiptonAuthors Info & Claims

Queue, Volume 16, Issue 3

Pages 31 - 57

https://doi.org/10.1145/3236386.3241340

Published: 01 June 2018 Publication History

All formats PDF

Abstract

Supervised machine-learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world?

References

[1]

Athey, S., Imbens, G. W. 2015 Machine-learning methods https://arxiv.org/abs/1504.01132v1 (see also ref. 7).

Google Scholar

[2]

Caruana, R., Kangarloo, H., Dionisio, J. D, Sinha, U., Johnson, D. 1999. Case-based explanation of non-case- based learning methods. In Proceedings of the American Medical Informatics Association (AMIA) Symposium: 212-215.

Google Scholar

[3]

Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21st Annual SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721-1730.

Digital Library

Google Scholar

[4]

Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., Blei, D. M. 2009. Reading tea leaves: how humans interpret topic models. In Proceedings of the 22nd International Conference on Neural Information Processing Systems (NIPS), 288-296.

Digital Library

Google Scholar

[5]

Doshi-Velez, F., Wallace, B., Adams, R. 2015. Graph- sparse lDA: a topic model with structured sparsity. In Proceedings of the 29th Association for the Advancement of Artificial Intelligence (AAAI) Conference, 2575-2581.

Digital Library

Google Scholar

[6]

FICO (Fair Isaac Corporation). 2011. Introduction to model builder scorecard; http://www.fico.com/en/latest-thinking/white-papers/introduction-to-model-builder-scorecard.

Google Scholar

[7]

Goodman, B., Flaxman, S. 2016. European Union regulations on algorithmic decision-making and a "right to explanation." https://arxiv.org/abs/1606.08813v3.

Google Scholar

[8]

Huysmans, J., Dejaeger, K., Mues, C., Vanthienen, J., Baesens, B. 2011. An empirical evaluation of the comprehensibility of decision table, tree- and rule- based predictive models. Journal of Decision Support Systems 51(1), 141-154.

Digital Library

Google Scholar

[9]

Kim, B. 2015. Interactive and interpretable machine- learning models for human-machine collaboration. Ph.D. thesis. Massachusetts Institute of Technology.

Google Scholar

[10]

Kim, B., Rudin, C., Shah, J. A. 2014. The Bayesian case model: A generative approach for case-based reasoning and prototype classification. In Proceedings of the 27th International Conference on Neural Information Processing Systems (NIPS), volume 2, 1952-1960.

Digital Library

Google Scholar

[11]

Kim, B., Glassman, E., Johnson, B., Shah, J. 2015. iBCM: Interactive Bayesian case model empowering humans via intuitive interaction. Massachusetts Institute of Technology, Cambridge, MA.

Google Scholar

[12]

Krening, S., Harrison, B., Feigh, K., Isbell, C., Riedl, M., Thomaz, A. 2017. Learning from explanations using sentiment and advice in RL. IEEE Transactions on Cognitive and Developmental Systems 9(1), 41-55.

Crossref

Google Scholar

[13]

Lipton, Z. C., Kale, D. C., Wetzel, R. 2016. Modeling missing data in clinical time series with RNNs. In Proceedings of Machine Learning for Healthcare.

Google Scholar

[14]

Liu, C., Rani, P., Sarkar, N. 2006. An empirical study of machine-learning techniques for affect recognition in human-robot interaction. Pattern Analysis and Applications 9(1): 58-69.

Digital Library

Google Scholar

[15]

Lou, Y., Caruana, R., Gehrke, J. 2012. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 150-158.

Digital Library

Google Scholar

[16]

Lou, Y., Caruana, R., Gehrke, J., Hooker, G. 2013. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 623-631.

Digital Library

Google Scholar

[17]

Mahendran, A., Vedaldi, A. 2015. Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1-9.

Crossref

Google Scholar

[18]

McAuley, J., Leskovec, J. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, 165-172.

Digital Library

Google Scholar

[19]

Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., Dean, J. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS), volume 2, 3111?3119.

Digital Library

Google Scholar

[20]

Mordvintsev, A., Olah, C., Tyka, M. 2015. Inceptionism: going deeper into neural networks. Google AI Blog; https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html.

Google Scholar

[21]

Mounk, Y. 2014. Is Harvard unfair to Asian-Americans? New York Times (Nov. 24); http://www.nytimes.com/2014/11/25/opinion/is-harvard-unfair-to-asian-americans.html.

Google Scholar

[22]

Pearl, J. 2009. Causality. Cambridge University Press.

Google Scholar

[23]

Ribeiro, M. T., Singh, S., Guestrin, C. 2016. "Why should I trust you?": explaining the predictions of any classifier. In Proceedings of the 22nd SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144.

Digital Library

Google Scholar

[24]

Ridgeway, G., Madigan, D., Richardson, T., O'Kane, J. 1998. Interpretable boosted naïve Bayes classification. In Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining: 101-104.

Digital Library

Google Scholar

[25]

Simonyan, K., Vedaldi, A., Zisserman, A. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. https://arxiv.org/abs/1312.6034 (see notes to refs 1, 7).

Google Scholar

[26]

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R. 2013. Intriguing properties of neural networks. https://arxiv.org/abs/1312.6199 (see refs 1, 7, 25).

Google Scholar

[27]

Tibshirani, R. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 58(1), 267-288.

Crossref

Google Scholar

[28]

Van der Maaten, L., Hinton, G. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579-2605.

Google Scholar

[29]

Wang, H.-X., Fratiglioni, L., Frisoni, G. B., Viitanen, M., Winblad, B. 1999. Smoking and the occurrence of Alzheimer's disease: cross-sectional and longitudinal data in a population-based study. American Journal of Epidemiology 149(7), 640-644.

Crossref

Google Scholar

[30]

Wang, Z., Freitas, N., Lanctot, M. 2016. Dueling network architectures for deep reinforcement learning. Proceedings of the 33rd International Conference on Machine Learning 48, 1995-2003.

Digital Library

Google Scholar

Cited By

View all

Fafoutellis PVlahogianni E(2025)A theory-informed multivariate causal framework for trustworthy short-term urban traffic forecastingTransportation Research Part C: Emerging Technologies10.1016/j.trc.2024.104945170(104945)Online publication date: Jan-2025
https://doi.org/10.1016/j.trc.2024.104945
Kim KKim JPark SLee JKim J(2025)A machine learning technique embedded reference-dependent choice model for explanatory power improvement: Shifting of reference point as a key factor in vehicle purchase decision-makingTransportation Research Part B: Methodological10.1016/j.trb.2024.103130191(103130)Online publication date: Jan-2025
https://doi.org/10.1016/j.trb.2024.103130
Tuo HMeng ZShi ZZhang D(2025)Interpretable neural network classification model using first-order logic rulesNeurocomputing10.1016/j.neucom.2024.128840614(128840)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128840
Show More Cited By

The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

Interpretability assessment of fuzzy knowledge bases

Computing with words (CWW) relies on linguistic representation of knowledge that is processed by operating at the semantical level defined through fuzzy sets. Linguistic representation of knowledge is a major issue when fuzzy rule based models are ...
Low-level interpretability and high-level interpretability: a unified view of data-driven interpretable fuzzy system modelling

This paper aims at providing an in-depth overview of designing interpretable fuzzy inference models from data within a unified framework. The objective of complex system modelling is to develop reliable and understandable models for human being to get ...
Improving the interpretability of classification rules discovered by an ant colony algorithm
GECCO '13: Proceedings of the 15th annual conference on Genetic and evolutionary computation

The vast majority of Ant Colony Optimization (ACO) algorithms for inducing classification rules use an ACO-based procedure to create a rule in an one-at-a-time fashion. An improved search strategy has been proposed in the cAnt-MinerPB algorithm, where ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

Queue Volume 16, Issue 3

Machine Learning

May-June 2018

118 pages

ISSN:1542-7730

EISSN:1542-7749

DOI:10.1145/3236386

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2018

Published in QUEUE Volume 16, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Popular
Editor picked

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,610
Total Citations
View Citations
81,507
Total Downloads

Downloads (Last 12 months)12,772
Downloads (Last 6 weeks)1,569

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fafoutellis PVlahogianni E(2025)A theory-informed multivariate causal framework for trustworthy short-term urban traffic forecastingTransportation Research Part C: Emerging Technologies10.1016/j.trc.2024.104945170(104945)Online publication date: Jan-2025
https://doi.org/10.1016/j.trc.2024.104945
Kim KKim JPark SLee JKim J(2025)A machine learning technique embedded reference-dependent choice model for explanatory power improvement: Shifting of reference point as a key factor in vehicle purchase decision-makingTransportation Research Part B: Methodological10.1016/j.trb.2024.103130191(103130)Online publication date: Jan-2025
https://doi.org/10.1016/j.trb.2024.103130
Tuo HMeng ZShi ZZhang D(2025)Interpretable neural network classification model using first-order logic rulesNeurocomputing10.1016/j.neucom.2024.128840614(128840)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128840
Lee JBaek KJeong HDoh SKim KCho K(2025)Revolutionizing cesium monitoring in seawater through electrochemical voltammetry and machine learningJournal of Hazardous Materials10.1016/j.jhazmat.2024.136558484(136558)Online publication date: Feb-2025
https://doi.org/10.1016/j.jhazmat.2024.136558
Mourali MNovakowski DPogacar RBrigden N(2025)Post hoc explanations improve consumer responses to algorithmic decisionsJournal of Business Research10.1016/j.jbusres.2024.114981186(114981)Online publication date: Jan-2025
https://doi.org/10.1016/j.jbusres.2024.114981
Prinzi FBarbiero PGreco CAmorese TCordasco GLiò PVitabile SEsposito A(2025)Using AI explainable models and handwriting/drawing tasks for psychological well-beingInformation Systems10.1016/j.is.2024.102465127(102465)Online publication date: Jan-2025
https://doi.org/10.1016/j.is.2024.102465
Escriva ELefrere TMartin MAligon JChanson AExcoffier JLabroche NSoulé-Dupuy CMonsarrat P(2025)Effective data exploration through clustering of local attributive explanationsInformation Systems10.1016/j.is.2024.102464127(102464)Online publication date: Jan-2025
https://doi.org/10.1016/j.is.2024.102464
Ong KMao RSatapathy RFilho RCambria ESulaeman JMengaldo G(2025)Explainable natural language processing for corporate sustainability analysisInformation Fusion10.1016/j.inffus.2024.102726115(102726)Online publication date: Mar-2025
https://doi.org/10.1016/j.inffus.2024.102726
He SXiao YMing AMa H(2025)Prompt-guided image color aesthetics assessment: Models, datasets and benchmarksInformation Fusion10.1016/j.inffus.2024.102706114(102706)Online publication date: Feb-2025
https://doi.org/10.1016/j.inffus.2024.102706
Zeng JChen KWang RLi YFan MWu KQi XWang L(2025)ContractMind: Trust-calibration interaction design for AI contract review toolsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103411196(103411)Online publication date: Feb-2025
https://doi.org/10.1016/j.ijhcs.2024.103411
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Magazine Site

View this article on the magazine site (external)

Magazine Site

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Recommendations

Interpretability assessment of fuzzy knowledge bases

Low-level interpretability and high-level interpretability: a unified view of data-driven interpretable fuzzy system modelling

Improving the interpretability of classification rules discovered by an ant colony algorithm

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Magazine Site

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations