More Web Proxy on the site http://driver.im/

research-article

EXS: Explainable Search Using Local Model Agnostic Interpretability

Authors:

Jaspreet Singh,

Avishek AnandAuthors Info & Claims

WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

Pages 770 - 773

https://doi.org/10.1145/3289600.3290620

Published: 30 January 2019 Publication History

Abstract

Retrieval models in information retrieval are used to rank documents for typically under-specified queries. Today machine learning is used to learn retrieval models from click logs and/or relevance judgments that maximizes an objective correlated with user satisfaction. As these models become increasingly powerful and sophisticated, they also become harder to understand. Consequently, it is hard for to identify artifacts in training, data specific biases and intents from a complex trained model like neural rankers even if trained purely on text features. EXS is a search system designed specifically to provide its users with insight into the following questions: "What is the intent of the query according to the ranker?'', "Why is this document ranked higher than another?'' and "Why is this document relevant to the query?''. EXS uses a version of a popular posthoc explanation method for classifiers -- LIME, adapted specifically to answer these questions. We show how such a system can effectively help a user understand the results of neural rankers and highlight areas of improvement.

References

[1]

David Alvarez-Melis and Tommi S Jaakkola. 2017. A causal framework for explaining the predictions of black-box sequence-to-sequence models. arXiv preprint arXiv:1707.01943 (2017).

[2]

Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. 2015. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1721--1730.

Digital Library

[3]

Suthee Chaidaroon and Yi Fang. 2017. Variational Deep Semantic Hashing for Text Documents. (2017).

[4]

Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016. Query expansion with locally-trained word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Vol. 1. Association for Computational Linguistics, 367--377.

[5]

Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. (2017).

[6]

Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, 55--64.

Digital Library

[7]

Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. arXiv preprint arXiv:1703.04730 (2017).

Digital Library

[8]

Benjamin Letham, Cynthia Rudin, Tyler H McCormick, David Madigan, and others. 2015. Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics, Vol. 9, 3 (2015), 1350--1371.

[9]

Bhaskar Mitra and Nick Craswell. 2017. Neural Models for Information Retrieval. arXiv preprint arXiv:1705.01509 (2017).

[10]

Bhaskar Mitra, Eric Nalisnick, Nick Craswell, and Rich Caruana. 2016. A dual embedding space model for document ranking. arXiv preprint arXiv:1602.01137 (2016).

[11]

Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. 2016. Improving document ranking with dual word embeddings. In Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 83--84.

Digital Library

[12]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016a. Model-agnostic interpretability of machine learning. arXiv preprint arXiv:1606.05386 (2016).

[13]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016b. Why Should I Trust You?: Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 1135--1144.

Digital Library

[14]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Anchors: High-precision model-agnostic explanations. In AAAI Conference on Artificial Intelligence .

[15]

Jaspreet Singh and Avishek Anand. 2018. Posthoc Interpretability of Learning to Rank Models using Secondary Training Data. SIGIR'18 Workshop on Explainable Search and Reccomendation, arXiv preprint arXiv:1806.11330 (2018).

[16]

Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017. IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models. arXiv preprint arXiv:1705.10513 (2017).

[17]

Adrian Weller. 2017. Challenges for Transparency. arXiv preprint arXiv:1708.01870 (2017).

[18]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning. 2048--2057.

Digital Library

[19]

Hamed Zamani and W Bruce Croft. 2016a. Embedding-based Query Language Models. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval. ACM, 147--156.

Digital Library

[20]

Hamed Zamani and W Bruce Croft. 2016b. Estimating Embedding Vectors for Queries. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval. ACM, 123--132.

Digital Library

Cited By

Xu ZLamba HAi QTetreault JJaimes AOosterhuis HBast HXiong C(2024)CFE2: Counterfactual Editing for Search Result ExplanationProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672508(145-155)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672508
Li HGu JLu XShen DLiu YDeng YShi GXiong H(2024)Beyond Relevance: Factor-level Causal Explanation for User Travel Decisions with Counterfactual Data AugmentationACM Transactions on Information Systems10.1145/365367342:5(1-31)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3653673
Chen CEickhoff CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Search System Explainability with Psychometrics and CrowdsourcingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657796(1051-1061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657796
Show More Cited By

Index Terms

EXS: Explainable Search Using Local Model Agnostic Interpretability
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
    2. Users and interactive retrieval
      1. Search interfaces

Recommendations

Towards Explainable Search Results: A Listwise Explanation Generator
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

It has been shown that the interpretability of search results is enhanced when query aspects covered by documents are explicitly provided. However, existing work on aspect-oriented explanation of search results explains each document independently. ...
Privacy-aware Document Ranking with Neural Signals
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

The recent work on neural ranking has achieved solid relevance improvement, by exploring similarities between documents and queries using word embeddings. It is an open problem how to leverage such an advancement for privacy-aware ranking, which is ...
From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management

The availability of massive data and computing power allowing for effective data driven neural approaches is having a major impact on machine learning and information retrieval research, but these models have a basic problem with efficiency. Current ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

January 2019

874 pages

ISBN:9781450359405

DOI:10.1145/3289600

General Chairs:
J. Shane Culpepper
RMIT University
,
Alistair Moffat
The University of Melbourne
,
Program Chairs:
Paul N. Bennett
Microsoft
,
Kristina Lerman
University of Southern California

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 January 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Research Council

Conference

WSDM '19

Sponsor:

WSDM '19: The Twelfth ACM International Conference on Web Search and Data Mining

February 11 - 15, 2019

Melbourne VIC, Australia

Acceptance Rates

WSDM '19 Paper Acceptance Rate 84 of 511 submissions, 16%;

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
789
Total Downloads

Downloads (Last 12 months)109
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu ZLamba HAi QTetreault JJaimes AOosterhuis HBast HXiong C(2024)CFE2: Counterfactual Editing for Search Result ExplanationProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672508(145-155)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672508
Li HGu JLu XShen DLiu YDeng YShi GXiong H(2024)Beyond Relevance: Factor-level Causal Explanation for User Travel Decisions with Counterfactual Data AugmentationACM Transactions on Information Systems10.1145/365367342:5(1-31)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3653673
Chen CEickhoff CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Search System Explainability with Psychometrics and CrowdsourcingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657796(1051-1061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657796
Golzadeh KGolab LSzlichta J(2024)Explaining Expert Search Systems with ExES2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00429(5465-5468)Online publication date: 13-May-2024
https://doi.org/10.1109/ICDE60146.2024.00429
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Conclusions and Open ChallengesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_6(143-146)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_6
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Privacy and SecurityTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_5(103-141)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_5
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)TransparencyTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_4(69-102)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_4
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Biases, Fairness, and Non-discriminationTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_3(29-67)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_3
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)Regulatory InitiativesTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_2(11-27)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_2
Schedl MAnelli VLex ESchedl MAnelli VLex E(2024)IntroductionTechnical and Regulatory Perspectives on Information Retrieval and Recommender Systems10.1007/978-3-031-69978-8_1(1-10)Online publication date: 24-Oct-2024
https://doi.org/10.1007/978-3-031-69978-8_1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents