More Web Proxy on the site http://driver.im/

article

Free access

Optimizing for Interpretability in Deep Neural Networks with Tree Regularization

Authors:

Sonali Parbhoo,

Michael C. Hughes,

Finale Doshi-VelezAuthors Info & Claims

Journal of Artificial Intelligence Research, Volume 72

Pages 1 - 37

https://doi.org/10.1613/jair.1.12558

Published: 04 January 2022 Publication History

Abstract

Deep models have advanced prediction in many domains, but their lack of interpretability remains a key barrier to the adoption in many real world applications. There exists a large body of work aiming to help humans understand these black box functions to varying levels of granularity – for example, through distillation, gradients, or adversarial examples. These methods however, all tackle interpretability as a separate process after training. In this work, we take a different approach and explicitly regularize deep models so that they are well-approximated by processes that humans can step through in little time. Specifically, we train several families of deep neural networks to resemble compact, axis-aligned decision trees without significant compromises in accuracy. The resulting axis-aligned decision functions uniquely make tree regularized models easy for humans to interpret. Moreover, for situations in which a single, global tree is a poor estimator, we introduce a regional tree regularizer that encourages the deep model to resemble a compact, axis-aligned decision tree in predefined, human-interpretable contexts. Using intuitive toy examples, benchmark image datasets, and medical tasks for patients in critical care and with HIV, we demonstrate that this new family of tree regularizers yield models that are easier for humans to simulate than L1 or L2 penalties without sacrificing predictive power.

References

[1]

Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016). Auditing black-box models for indirect influence. In <italic>ICDM</italic>.

[2]

Amir, D., & Amir, O. (2018). Highlights: Summarizing agent behavior to people. In <italic>Proc. of the 17th International conference on Autonomous Agents and Multi-Agent Systems (AAMAS)</italic>.

Digital Library

[3]

Audet, C., & Kokkolaras, M. (2016). <italic>Blackbox and derivative-free optimization: theory, algorithms and applications</italic>. Springer.

[4]

Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. <italic>PloS one, 10</italic> (7), e0130140.

[5]

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. <italic>arXiv preprint arXiv:1409.0473</italic>.

[6]

Balan, A. K., Rathod, V., Murphy, K. P., & Welling, M. (2015). Bayesian dark knowledge. In <italic>NIPS</italic>.

Digital Library

[7]

Balestriero, R. (2017). Neural decision trees. <italic>arXiv preprint arXiv:1702.07360</italic>.

[8]

Binder, A., Bach, S., Montavon, G., Müller, K.-R., & Samek, W. (2016). Layer-wise relevance propagation for deep neural network architectures. In <italic>Information Science and Applications (ICISA) 2016</italic>, pp. 913-922. Springer.

[9]

Bucilua, C., Caruana, R., & Niculescu-Mizil, A. (2006). Model compression. In <italic>KDD</italic>.

Digital Library

[10]

Che, Z., Kale, D., Li, W., Bahadori, M. T., & Liu, Y. (2015). Deep computational phenotyping. In <italic>KDD</italic>.

Digital Library

[11]

Chen, J. H., Asch, S. M., et al. (2017). Machine learning and prediction in medicine-beyond the peak of inflated expectations. <italic>N Engl J Med, 376</italic> (26), 2507-2509.

[12]

Cho, K., Gulcehre, B. v. M. C., Bahdanau, D., Schwenk, F. B. H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In <italic>EMLNP</italic>.

[13]

Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2016). Doctor AI: Predicting clinical events via recurrent neural networks. In <italic>Machine Learning for Healthcare Conference</italic>.

[14]

Craven, M., & Shavlik, J. W. (1996). Extracting tree-structured representations of trained networks. In <italic>Advances in neural information processing systems</italic>, pp. 24-30.

Digital Library

[15]

Dheeru, D., & Karra Taniskidou, E. (2017). UCI machine learning repository.

[16]

Drucker, H., & Le Cun, Y. (1992). Improving generalization performance using double backpropagation. <italic>IEEE Transactions on Neural Networks, 3</italic> (6), 991-997.

Digital Library

[17]

Duchi, J., Shalev-Shwartz, S., Singer, Y., & Chandra, T. (2008). Efficient projections onto the l 1-ball for learning in high dimensions. In <italic>Proceedings of the 25th international conference on Machine learning</italic>, pp. 272-279. ACM.

Digital Library

[18]

Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network. Tech. rep. 1341, Department of Computer Science and Operations Research, University of Montreal.

[19]

Frosst, N., & Hinton, G. (2017). Distilling a neural network into a soft decision tree. <italic>arXiv preprint arXiv:1711.09784</italic>.

[20]

Garofolo, J. S., et al. (1993). TIMIT acoustic-phonetic continuous speech corpus. <italic>Linguistic Data Consortium, 10</italic> (5).

[21]

Ghassemi, M., Wu, M., Hughes, M. C., Szolovits, P., & Doshi-Velez, F. (2017). Predicting intervention onset in the icu with switching state space models. <italic>AMIA Summits on Translational Science Proceedings, 2017</italic>, 82.

[22]

Goodfellow, I., Bengio, Y., & Courville, A. (2016). <italic>Deep Learning</italic>. MIT Press.

Digital Library

[23]

Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. <italic>Jama, 316</italic> (22), 2402-2410.

[24]

Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. In <italic>NIPS</italic>.

Digital Library

[25]

Hendrycks, D., & Gimpel, K. (2016). Gaussian error linear units (gelus). <italic>arXiv preprint arXiv:1606.08415</italic>.

[26]

Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. <italic>arXiv preprint arXiv:1503.02531</italic>.

[27]

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. <italic>Neural computation</italic>, <italic>9</italic> (8), 1735-1780.

Digital Library

[28]

Hu, Z., Ma, X., Liu, Z., Hovy, E., & Xing, E. (2016). Harnessing deep neural networks with logic rules. In <italic>ACL</italic>.

[29]

Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. <italic>arXiv preprint arXiv:1502.03167</italic>.

Digital Library

[30]

Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L. A., & Mark, R. G. (2016). MIMIC-III, a freely accessible critical care database. <italic>Scientific Data, 3</italic>.

[31]

Joliffe, I. T., & Morgan, B. (1992). Principal component analysis and exploratory factor analysis. <italic>Statistical methods in medical research, 1</italic> (1), 69-95.

[32]

Kim, B., Rudin, C., & Shah, J. A. (2014). The bayesian case model: A generative approach for case-based reasoning and prototype classification. In <italic>Advances in Neural Information Processing Systems</italic>, pp. 1952-1960.

Digital Library

[33]

Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. <italic>arXiv preprint arXiv:1412.6980</italic>.

[34]

Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. <italic>arXiv preprint arXiv:1703.04730</italic>.

Digital Library

[35]

Kohavi, R. (1996). Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. In <italic>KDD</italic>, Vol. 96, pp. 202-207. Citeseer.

Digital Library

[36]

Kontschieder, P., Fiterau, M., Criminisi, A., & Bulo, S. R. (2015). Deep neural decision forests. In <italic>Proceedings of the IEEE international conference on computer vision</italic>, pp. 1467-1475.

Digital Library

[37]

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In <italic>Advances in neural information processing systems</italic>, pp. 1097-1105.

Digital Library

[38]

Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016). Interpretable decision sets: A joint framework for description and prediction. In <italic>Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining</italic>, pp. 1675-1684. ACM.

Digital Library

[39]

LeCun, Y. (1998). The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/.

[40]

Lei, T., Barzilay, R., & Jaakkola, T. (2016). Rationalizing neural predictions. <italic>arXiv preprint arXiv:1606.04155</italic>.

[41]

Lipton, Z. C. (2016). The mythos of model interpretability. In <italic>ICML Workshop on Human Interpretability in Machine Learning</italic>.

[42]

Lundberg, S., & Lee, S.-I. (2016). An unexpected unity among methods for interpreting model predictions. <italic>arXiv preprint arXiv:1611.07478</italic>.

[43]

Maaten, L. v. d., & Hinton, G. (2008). Visualizing data using t-sne. <italic>Journal of machine learning research, 9</italic>(Nov), 2579-2605.

[44]

Martins, A., & Astudillo, R. (2016). From softmax to sparsemax: A sparse model of attention and multi-label classification. In <italic>International Conference on Machine Learning</italic>, pp. 1614-1623.

Digital Library

[45]

Miller, T. (2018). Explanation in artificial intelligence: Insights from the social sciences. <italic>Artificial Intelligence</italic>.

[46]

Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016). Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. <italic>Scientific reports, 6</italic>, 26094.

[47]

Montavon, G., Samek, W., & Müller, K.-R. (2018). Methods for interpreting and understanding deep neural networks. <italic>Digital Signal Processing, 73</italic>, 1-15.

[48]

Mordvintsev, A., Olah, C., & Tyka, M. (2015). Inceptionism: Going deeper into neural networks. <italic>Google Research Blog. Retrieved June, 20</italic> (14), 5.

[49]

Moro, S., Cortez, P., & Rita, P. (2014). A data-driven approach to predict the success of bank telemarketing. <italic>Decision Support Systems, 62</italic>, 22-31.

[50]

Nguyen, A., Yosinski, J., & Clune, J. (2016). Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. <italic>arXiv preprint arXiv:1602.03616</italic>.

[51]

Nguyen, T. D., Kasmarik, K. E., & Abbass, H. A. (2020). Towards interpretable deep neural networks: An exact transformation to multi-class multivariate decision trees. <italic>arXiv e-prints</italic>, arXiv-2003.

[52]

Ochiai, T., Matsuda, S., Watanabe, H., & Katagiri, S. (2017). Automatic node selection for deep neural networks using group lasso regularization. In <italic>ICASSP</italic>.

[53]

Organization, W. H., et al. (2005). Interim who clinical staging of hvi/aids and hiv/aids case definitions for surveillance: African region. Tech. rep., Geneva: World Health Organization.

[54]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011a). Scikit-learn: Machine learning in Python. <italic>Journal of Machine Learning Research, 12</italic>, 2825-2830.

Digital Library

[55]

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al. (2011b). Scikit-learn: Machine learning in python. <italic>Journal of machine learning research, 12</italic> (Oct), 2825-2830.

Digital Library

[56]

Rabiner, L., & Juang, B. (1986). An introduction to hidden markov models. <italic>ieee assp magazine, 3</italic> (1), 4-16.

[57]

Rastegari, M., Ordonez, V., Redmon, J., & Farhadi, A. (2016). XNOR-Net: ImageNet classification using binary convolutional neural networks. In <italic>ECCV</italic>.

[58]

Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In <italic>Advances in neural information processing systems</italic>, pp. 91-99.

Digital Library

[59]

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Why should i trust you?: Explaining the predictions of any classifier. In <italic>Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining</italic>, pp. 1135-1144. ACM.

Digital Library

[60]

Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017). Right for the right reasons: Training differentiable models by constraining their explanations. <italic>arXiv preprint arXiv:1703.03717</italic>.

[61]

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. <italic>arXiv preprint arXiv:1610.02391v3</italic>.

[62]

Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016). Grad-cam: Why did you say that?. <italic>arXiv preprint arXiv:1611.07450</italic>.

[63]

Simonyan, K., Vedaldi, A., & Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. <italic>arXiv preprint arXiv:1312.6034</italic>.

[64]

Singh, S., Ribeiro, M. T., & Guestrin, C. (2016). Programs as black-box explanations. <italic>arXiv preprint arXiv:1611.07579</italic>.

[65]

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In <italic>NIPS</italic>.

Digital Library

[66]

Tang, W., Hua, G., & Wang, L. (2017). How to train a compact binary neural network with high accuracy?. In <italic>AAAI</italic>.

Digital Library

[67]

Wan, A., Dunlap, L., Ho, D., Yin, J., Lee, S., Jin, H., Petryk, S., Bargal, S. A., & Gonzalez, J. E. (2020). Nbdt: neural-backed decision trees. <italic>arXiv preprint arXiv:2004.00221</italic>.

[68]

Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. <italic>arXiv preprint arXiv:1708.07747</italic>.

[69]

Yang, Y., Morillo, I. G., & Hospedales, T. M. (2018). Deep neural decision trees. <italic>arXiv preprint arXiv:1806.06988</italic>.

[70]

Zazzi, M., Incardona, F., Rosen-Zvi, M., Prosperi, M., Lengauer, T., Altmann, A., Sonnerborg, A., Lavee, T., Schulter, E., & Kaiser, R. (2012). Predicting response to antiretroviral treatment by machine learning: the euresist project. <italic>Intervirology, 55</italic> (2), 123-127.

[71]

Zhang, Y., Lee, J. D., & Jordan, M. I. (2016). l1-regularized neural networks are improperly learnable in polynomial time. In <italic>ICML</italic>.

Digital Library

Cited By

Lu BQuan C(2023)Enhancing Innovation Management and Venture Capital Evaluation via Advanced Deep Learning TechniquesJournal of Organizational and End User Computing10.4018/JOEUC.33508136:1(1-22)Online publication date: 18-Dec-2023
https://dl.acm.org/doi/10.4018/JOEUC.335081
Baier AAspandi DStaab SElkind E(2023)ReLiNetProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/385(3461-3469)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/385
Chattopadhyay ASlocum SHaeffele BVidal RGeman D(2023)Interpretable by Design: Learning Predictors by Composing Interpretable QueriesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.322516245:6(7430-7443)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3225162
Show More Cited By

Recommendations

Revisiting multiple instance neural networks

We revisit the problem of solving MIL using neural networks (MINNs), which are ignored in current MIL research community. Our experiments show that MINNs are very effective and efficient.We proposed a novel MI-Net which is centered on learning bag ...
Adapting RBF Neural Networks to Multi-Instance Learning

In multi-instance learning, the training examples are bags composed of instances without labels, and the task is to predict the labels of unseen bags through analyzing the training bags with known labels. A bag is positive if it contains at least one ...
On mapping decision trees and neural networks

There exist several methods for transforming decision trees to neural networks. These methods typically construct the networks by directly mapping decision nodes or rules to the neural units. As a result, the networks constructed are often larger than ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Artificial Intelligence Research

Journal of Artificial Intelligence Research Volume 72, Issue

Jan 2022

1485 pages

ISSN:1076-9757

Issue’s Table of Contents

Publisher

AI Access Foundation

El Segundo, CA, United States

Publication History

Published: 04 January 2022

Published in JAIR Volume 72

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
353
Total Downloads

Downloads (Last 12 months)99
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lu BQuan C(2023)Enhancing Innovation Management and Venture Capital Evaluation via Advanced Deep Learning TechniquesJournal of Organizational and End User Computing10.4018/JOEUC.33508136:1(1-22)Online publication date: 18-Dec-2023
https://dl.acm.org/doi/10.4018/JOEUC.335081
Baier AAspandi DStaab SElkind E(2023)ReLiNetProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/385(3461-3469)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/385
Chattopadhyay ASlocum SHaeffele BVidal RGeman D(2023)Interpretable by Design: Learning Predictors by Composing Interpretable QueriesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.322516245:6(7430-7443)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3225162
Chen SGao CZhang P(2022)Incorporation of Data-Mined Knowledge into Black-Box SVM for InterpretabilityACM Transactions on Intelligent Systems and Technology10.1145/354877514:1(1-22)Online publication date: 9-Nov-2022
https://dl.acm.org/doi/10.1145/3548775

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents