More Web Proxy on the site http://driver.im/

research-article

Analyzing and Learning from User Interactions for Search Clarification

Authors:

Paul N. Bennett,

Susan T. DumaisAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1181 - 1190

https://doi.org/10.1145/3397271.3401160

Published: 25 July 2020 Publication History

Abstract

Asking clarifying questions in response to search queries has been recognized as a useful technique for revealing the underlying intent of the query. Clarification has applications in retrieval systems with different interfaces, from the traditional web search interfaces to the limited bandwidth interfaces as in speech-only and small screen devices. Generation and evaluation of clarifying questions have been recently studied in the literature. However, user interaction with clarifying questions is relatively unexplored. In this paper, we conduct a comprehensive study by analyzing large-scale user interactions with clarifying questions in a major web search engine. In more detail, we analyze the user engagements received by clarifying questions based on different properties of search queries, clarifying questions, and their candidate answers. We further study click bias in the data, and show that even though reading clarifying questions and candidate answers does not take significant efforts, there still exist some position and presentation biases in the data. We also propose a model for learning representation for clarifying questions based on the user interaction data as implicit feedback. The model is used for re-ranking a number of automatically generated clarifying questions for a given query. Evaluation on both click data and human labeled data demonstrates the high quality of the proposed method.

References

[1]

Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky, and Marc Najork. 2019. Addressing Trust Bias for Unbiased Learning-to-Rank. In WWW '19. 4--14.

[2]

Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019. Asking Clarifying Questions in Open-Domain Information-Seeking Conversations. In SIGIR '19. 475--484.

[3]

James Allan. 2004. HARD Track Overview in TREC 2004: High Accuracy Retrieval from Documents. In TREC '04.

[4]

Nicholas J. Belkin, Colleen Cool, Adelheit Stein, and Ulrich Thiel. 1995. Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems. Expert Systems with Applications, Vol. 9, 3 (1995), 379--395.

[5]

Paul N. Bennett, Ryen W. White, Wei Chu, Susan T. Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012. Modeling the Impact of Short- and Long-term Behavior on Search Personalization. In SIGIR '12. 185--194.

[6]

Marco De Boni and Suresh Manandhar. 2003. An Analysis of Clarification Dialogue for Question Answering. In NAACL '03. 48--55.

[7]

Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What Do You Mean Exactly?: Analyzing Clarification Questions in CQA. In CHIIR '17. 345--348.

Digital Library

[8]

Christopher J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An Overview . Technical Report. Microsoft Research.

[9]

Fei Cai and Maarten de Rijke. 2016. A Survey of Query Auto Completion in Information Retrieval. Now Publishers Inc., Hanover, MA, USA.

[10]

Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards Conversational Recommender Systems. In KDD '16. 815--824.

[11]

Anni Coden, Daniel Gruhl, Neal Lewis, and Pablo N. Mendes. 2015. Did you mean A or B? Supporting Clarification Dialog for Entity Disambiguation. In SumPre '15.

[12]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An Experimental Comparison of Click Position-Bias Models. In WSDM '08. 87--94.

[13]

Marco De Boni and Suresh Manandhar. 2005. Implementing Clarification Dialogues in Open Domain Question Answering. Nat. Lang. Eng., Vol. 11, 4 (2005), 343--361.

Digital Library

[14]

Mostafa Dehghani, Sascha Rothe, Enrique Alfonseca, and Pascal Fleury. 2017. Learning to Attend, Copy, and Generate for Session-Based Query Suggestion. In CIKM '17. 1747--1756.

Digital Library

[15]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL '19. Minneapolis, Minnesota, 4171--4186.

[16]

Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying Relations for Open Information Extraction. In EMNLP '11. 1535--1545.

[17]

Manish Gupta and Michael Bendersky. 2015. Information Retrieval with Verbose Queries. Foundations and Trends® in Information Retrieval, Vol. 9, 3--4 (2015), 209--354.

Digital Library

[18]

Helia Hashemi, Hamed Zamani, and W. Bruce Croft. 2020. Guided Transformer: Leveraging Multiple External Sources for Representation Learning in Conversational Search. In SIGIR '20.

[19]

Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey. 2013. Beyond clicks: query reformulation as a predictor of search satisfaction. In CIKM '13. 2019--2028.

Digital Library

[20]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data as Implicit Feedback. In SIGIR '05. 154--161.

Digital Library

[21]

Eugene Kharitonov and Pavel Serdyukov. 2012. Demographic Context in Web Search Re-ranking. In CIKM '12. 2555--2558.

[22]

Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. 2018. Toward Voice Query Clarification. In SIGIR '18. 1257--1260.

[23]

Weize Kong, Rui Li, Jie Luo, Aston Zhang, Yi Chang, and James Allan. 2015. Predicting Search Intent Based on Pre-Search Context. In SIGIR '15. 503--512.

[24]

Danai Koutra, Paul N. Bennett, and Eric Horvitz. 2015. Events and Controversies: Influences of a Shocking News Event on Information Seeking. In WWW '15 . 614--624.

Digital Library

[25]

Widad Machmouchi and Georg Buscher. 2016. Principles for the Design of Online A/B Metrics. SIGIR '16. 589--590.

Digital Library

[26]

Nicolaas Matthijs and Filip Radlinski. 2011. Personalizing Web Search Using Long Term Browsing History. In WSDM '11. 25--34.

[27]

Bhaskar Mitra, Milad Shokouhi, Filip Radlinski, and Katja Hofmann. 2014. On User Interactions with Query Auto-Completion. SIGIR '14. 1055--1058.

[28]

Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating Natural Questions About an Image. In ACL '16. Berlin, Germany, 1802--1813.

[29]

Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. (2019). arxiv: cs.IR/1901.04085

[30]

Umut Ozertem, Olivier Chapelle, Pinar Donmez, and Emre Velipasaoglu. 2012. Learning to Suggest: A Machine Learning Framework for Ranking Query Suggestions. In SIGIR '12. 25--34.

Digital Library

[31]

Neil O'Hare, Paloma de Juan, Rossano Schifanella, Yunlong He, Dawei Yin, and Yi Chang. 2016. Leveraging User Interaction Signals for Web Image Search. In SIGIR '16. 559--568.

[32]

Harshith Padigela, Hamed Zamani, and W. Bruce Croft. 2019. Investigating the Successes and Failures of BERT for Passage Re-Ranking. (2019). arxiv: cs.IR/1905.01758

[33]

Luis Quintano and Irene Pimenta Rodrigues. 2008. Question/Answering Clarification Dialogues. In MICAI '08. 155--164.

[34]

Filip Radlinski and Nick Craswell. 2017. A Theoretical Framework for Conversational Search. CHIIR '17. 117--126.

Digital Library

[35]

Sudha Rao and Hal Daumé III. 2018. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. ACL '18. Melbourne, Australia, 2737--2746.

[36]

Sudha Rao and Hal Daumé III. 2019. Answer-based Adversarial Training for Generating Clarification Questions. In NAACL '19. Minneapolis, Minnesota.

[37]

Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting Clicks: Estimating the Click-through Rate for New Ads. In WWW '07. 521--530.

Digital Library

[38]

Mark Sanderson. 2008. Ambiguous Queries: Test Collections Need More Sense. In SIGIR '08. 499--506.

Digital Library

[39]

Rodrygo L. Santos, Craig Macdonald, and Iadh Ounis. 2013. Learning to Rank Query Suggestions for Adhoc and Diversity Search. Inf. Retr., Vol. 16, 4 (2013), 429--451.

Digital Library

[40]

Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2015. Search Result Diversification. Found. Trends Inf. Retr., Vol. 9, 1 (2015), 1--90.

Digital Library

[41]

Milad Shokouhi. 2013. Learning to Personalize Query Auto-Completion. In SIGIR '13. 103--112.

Digital Library

[42]

Svetlana Stoyanchev, Alex Liu, and Julia Hirschberg. 2014. Towards Natural Clarification Questions in Dialogue Systems. In AISB '14, Vol. 20.

[43]

Yueming Sun and Yi Zhang. 2018. Conversational Recommender System. In SIGIR '18. 235--244.

[44]

Jan Trienes and Krisztian Balog. 2019. Identifying Unclear Questions in Community Question Answering Websites. In ECIR '19. 276--289.

[45]

Yury Ustinovskiy and Pavel Serdyukov. 2013. Personalization of web-search using short-term browsing context. In CIKM '13. 1979--1988.

Digital Library

[46]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In NeurIPS '17. 6000--6010.

[47]

Xiaohui Xie, Jiaxin Mao, Maarten de Rijke, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2018. Constructing an Interaction Behavior Model for Web Image Search. In SIGIR '18. 425--434.

Digital Library

[48]

Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond Position Bias: Examining Result Attractiveness as a Source of Presentation Bias in Clickthrough Data. In WWW '10. 1011--1018.

Digital Library

[49]

Hamed Zamani, Michael Bendersky, Xuanhui Wang, and Mingyang Zhang. 2017. Situational Context for Ranking in Personal Search. In WWW '17. 1531--1540.

[50]

Hamed Zamani and Nick Craswell. 2020. Macaw: An Extensible Conversational Information Seeking Platform. In SIGIR '20.

[51]

Hamed Zamani, Susan T. Dumais, Nick Craswell, Paul N. Bennett, and Gord Lueck. 2020. Generating Clarifying Questions for Information Retrieval. In WWW '20. 418--428.

[52]

Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018. Towards Conversational Search and Recommendation: System Ask, User Respond. In CIKM '18. 177--186.

Cited By

Tavakoli LTrippas JZamani HScholer FSanderson M(2024)Online and Offline Evaluation in Search ClarificationACM Transactions on Information Systems10.1145/368178643:1(1-30)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3681786
Zhao ZDou ZZhou YSerra ESpezzano F(2024)Generating Intent-aware Clarifying Questions in Conversational Information Retrieval SystemsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679851(3384-3394)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679851
Samarinas CZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ProCIS: A Benchmark for Proactive Retrieval in ConversationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657869(830-840)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657869
Show More Cited By

Index Terms

Analyzing and Learning from User Interactions for Search Clarification
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing
      1. Query intent
      2. Query log analysis
    2. Specialized information retrieval
      1. Environment-specific retrieval
        Web and social media search

Recommendations

Identifying popular search goals behind search queries to improve web search ranking
AIRS'11: Proceedings of the 7th Asia conference on Information Retrieval Technology

Web users usually have a certain search goal before they submit a search query. However, many laypersons can't transform their search goals into suitable queries. Thus, understanding original search goals behind a query is very important for search ...
Determining the user intent behind web search queries by learning from past user interactions with search results
COMAD '13: Proceedings of the 19th International Conference on Management of Data

Understanding the intent of users behind their web search queries is very useful in serving them with relevant advertisements and in ranking the search results effectively. Existing approaches broadly classify the user intent behind web queries into ...
Improving Ranking Consistency for Web Search by Leveraging a Knowledge Base and Search Logs
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

In this paper, we propose a new idea called ranking consistency in web search. Relevance ranking is one of the biggest problems in creating an effective web search system. Given some queries with similar search intents, conventional approaches typically ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
550
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)9

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tavakoli LTrippas JZamani HScholer FSanderson M(2024)Online and Offline Evaluation in Search ClarificationACM Transactions on Information Systems10.1145/368178643:1(1-30)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3681786
Zhao ZDou ZZhou YSerra ESpezzano F(2024)Generating Intent-aware Clarifying Questions in Conversational Information Retrieval SystemsProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679851(3384-3394)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679851
Samarinas CZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ProCIS: A Benchmark for Proactive Retrieval in ConversationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657869(830-840)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657869
Gienapp LScells HDeckers NBevendorff JWang SKiesel JSyed SFröbe MZuccon GStein BHagen MPotthast MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Generative Ad Hoc Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657849(1916-1929)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657849
Zhao ZDou ZChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Generating Multi-turn Clarification for Web Information SeekingProceedings of the ACM Web Conference 202410.1145/3589334.3645712(1539-1548)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645712
Wang ZXu ZSrikumar VAi QChua TNgo CKa-Wei Lee RKumar RLauw H(2024)An In-depth Investigation of User Response Simulation for Conversational SearchProceedings of the ACM Web Conference 202410.1145/3589334.3645447(1407-1418)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645447
Beelen TOrdelman RTruong KEvers VHuibers T(2024)Does conversation lead to better searches? Investigating single-shot and multi-turn spoken searches with childrenInternational Journal of Child-Computer Interaction10.1016/j.ijcci.2024.10066841(100668)Online publication date: Sep-2024
https://doi.org/10.1016/j.ijcci.2024.100668
Bedi NSekulić ICrestani F(2024)Clarifying Questions Generation for Conversational Search Based on “People Also Ask” FeatureSustainability and Empowerment in the Context of Digital Libraries10.1007/978-981-96-0865-2_20(246-260)Online publication date: 6-Dec-2024
https://doi.org/10.1007/978-981-96-0865-2_20
Sekulić IŁajewska WBalog KCrestani F(2024)Estimating the Usefulness of Clarifying Questions and Answers for Conversational SearchAdvances in Information Retrieval10.1007/978-3-031-56063-7_30(384-392)Online publication date: 23-Mar-2024
https://doi.org/10.1007/978-3-031-56063-7_30
Zhao ZDou ZGuo YCao ZCheng XSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Improving Search Clarification with Structured Information Extracted from Search ResultsProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599389(3549-3558)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599389
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents