[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3340531.3412017acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Providing Direct Answers in Search Results: A Study of User Behavior

Published: 19 October 2020 Publication History

Abstract

To study the impact of providing direct answers in search results on user behavior, we conducted a controlled user study to analyze factors including reading time, eye-tracked attention, and the influence of the quality of answer module content. We also studied a more advanced answer interface, where multiple answers are shown on the search engine results page (SERP). Our results show that users focus more extensively than normal on the top items in the result list when answers are provided. The existence of the answer module helps to improve user engagement on SERPs, reduces user effort, and promotes user satisfaction during the search process. Furthermore, we investigate how the question type -- factoid or non-factoid -- affects user interaction patterns. This work provides insight into the design of SERPs that includes direct answers to queries, including when answers should be shown.

Supplementary Material

MP4 File (3340531.3412017.mp4)
To study the impact of providing direct answers in search results on user behavior, we conducted a controlled user study to analyze factors including reading time, eye-tracked attention, and the influence of the quality of answer module content. We also studied a more advanced answer interface, where multiple answers are shown on the search engine results page (SERP). Our results show that users focus more extensively than normal on the top items in the result list when answers are provided. The existence of the answer module helps to improve user engagement on SERPs, reduces user effort, and promotes user satisfaction during the search process. Furthermore, we investigate how the question type ? factoid or non-factoid ? affects user interaction patterns. This work provides insight into the design of SERPs that includes direct answers to queries, including when answers should be shown.

References

[1]
Jaime Arguello, Fernando Diaz, Jamie Callan, and Jean-Francois Crespo. 2009. Sources of Evidence for Vertical Selection. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Boston, MA, USA, 315--322.
[2]
Anne Aula, P"aivi Majaranta, and Kari-Jouko R"aih"a. 2005. Eye-tracking reveals the personal styles for search result evaluation. In IFIP conference on human-computer interaction. 1058--1061.
[3]
Flavio Chierichetti, Ravi Kumar, and Prabhakar Raghavan. 2011. Optimizing Two-Dimensional Search Results Presentation. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining. 257--266.
[4]
Charles L. A. Clarke, Eugene Agichtein, Susan Dumais, and Ryen W. White. 2007. The Influence of Caption Features on Clickthrough Patterns in Web Search. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Amsterdam, The Netherlands, 135--142.
[5]
Michael J. Cole, Chathra Hendahewa, Nicholas J. Belkin, and Chirag Shah. 2015. User Activity Patterns During Information Search. ACM Trans. Inf. Syst. (2015).
[6]
N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. 2008. An experimental comparison of click position-bias models. In Proceedings of the international conference on web search and web data mining. Palo Alto, CA, 87--94.
[7]
Edward Cutrell and Zhiwei Guan. 2007. What are you looking for? An eye-tracking study of information usage in Web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 407--416.
[8]
Andrew Duchowski. 2007. Eye Tracking Methodology Theory and Practice 2 ed.). Springer.
[9]
Susan T. Dumais, Georg Buscher, and Edward Cutrell. 2010. Individual Differences in Gaze Patterns for Web Search. In Proceedings of the Third Symposium on Information Interaction in Context. New Brunswick, New Jersey, USA, 185--194.
[10]
Olive Jean Dunn. 1964. Multiple comparisons using rank sums. Technometrics, Vol. 6, 3 (1964), 241--252.
[11]
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, Vol. 76, 5 (1971), 378.
[12]
Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-Tracking Analysis of User Behavior in WWW Search. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
[13]
Jimmy Jimmy, Guido Zuccon, Bevan Koopman, and Gianluca Demartini. 2019. Health Cards for Consumer Health Search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery.
[14]
Thorsten Joachims, Laura A. Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval. Salvador, Brazil, 154--161.
[15]
Yvonne Kammerer and Peter Gerjets. 2014. The Role of Search Result Position and Source Trustworthiness in the Selection of Web Search Results When Using a List or a Grid Interface. International Journal of Human-Computer Interaction, Vol. 30, 3 (2014), 177--191.
[16]
Diane Kelly. 2009. Methods for evaluating interactive information retrieval systems with users. Foundations and trends in Information Retrieval (2009).
[17]
William H Kruskal and W Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, Vol. 47, 260 (1952).
[18]
Dmitry Lagun and Eugene Agichtein. 2011. ViewSer: Enabling Large-Scale Remote User Studies of Web Search Examination and Interaction. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. Beijing, China, 365--374.
[19]
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.
[20]
Xiangsheng Li, Yiqun Liu, Jiaxin Mao, Zexue He, Min Zhang, and Shaoping Ma. 2018. Understanding Reading Attention Distribution during Relevance Judgement. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery.
[21]
Yiqun Liu, Chao Wang, Ke Zhou, Jianyun Nie, Min Zhang, and Shaoping Ma. 2014. From Skimming to Reading: A Two-Stage Examination Model for Web Search. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (CIKM '14).
[22]
Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of Vertical Result in Web Search Examination. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '15). Association for Computing Machinery, New York, NY, USA, 193--202.
[23]
Lori Lorigo, Maya Haridasan, Hrönn Brynjarsdóttir, Ling Xia, Thorsten Joachims, Geri Gay, Laura Granka, Fabio Pellacini, and Bing Pan. 2008. Eye tracking and online search: Lessons learned and challenges ahead. JASIST, Vol. 59 (05 2008).
[24]
Emma Lurie and Eni Mustafaraj. 2018. Investigating the Effects of Google's Search Engine Result Page in Evaluating the Credibility of Online News Sources. In Proceedings of the 10th ACM Conference on Web Science.
[25]
Scott A McDonald and Richard C Shillcock. 2003. Low-level predictive inference in reading: The influence of transitional probabilities on eye movements. Vision Research, Vol. 43, 16 (2003), 1735--1751.
[26]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoRR, Vol. abs/1611.09268 (2016). arxiv: 1611.09268
[27]
Neil O'Hare, Paloma de Juan, Rossano Schifanella, Yunlong He, Dawei Yin, and Yi Chang. 2016. Leveraging User Interaction Signals for Web Image Search. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '16). 10.
[28]
Bing Pan, Helene A. Hembrooke, Geri K. Gay, Laura A. Granka, Matthew K. Feusner, and Jill K. Newman. 2004. The Determinants of Web Page Viewing Behavior: An Eye-Tracking Study. In Proceedings of the 2004 Symposium on Eye Tracking Research & Applications (ETRA '04). 147--154.
[29]
Chen Qu, Liu Yang, W Bruce Croft, Falk Scholer, and Yongfeng Zhang. 2019. Answer interaction in non-factoid question answering systems. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. ACM.
[30]
Daniel E. Rose and Danny Levinson. 2004. Understanding User Goals in Web Search. In Proceedings of the 13th International Conference on World Wide Web.
[31]
C James Scheirer, William S Ray, and Nathan Hare. 1976. The analysis of ranked data derived from completely randomized factorial designs. Biometrics (1976).
[32]
Aliaksei Severyn and Alessandro Moschitti. 2013. Automatic feature engineering for answer selection and extraction. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 458--467.
[33]
Shanu Sushmita, Hideo Joho, Mounia Lalmas, and Robert Villa. 2010. Factors Affecting Click-through Behavior in Aggregated Search Interfaces. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM '10). 519--528.
[34]
Benjamin W. Tatler and Benjamin T. Vincent. 2009. The prominence of behavioural biases in eye guidance. Visual Cognition, Vol. 17, 6--7 (2009), 1029--1054.
[35]
Jaime Teevan, Edward Cutrell, Danyel Fisher, Steven M. Drucker, Gonzalo Ramos, Paul André, and Chang Hu. 2009. Visual Snippets: Summarizing Web Pages for Search and Revisitation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Boston, MA, USA, 2023--2032.
[36]
Paul Thomas, Alistair Moffat, Peter Bailey, Falk Scholer, and Nick Craswell. 2018. Better Effectiveness Metrics for SERPs, Cards, and Rankings. In Proceedings of the 23rd Australasian Document Computing Symposium. Dunedin, New Zealand, 8.
[37]
Paul Thomas, Falk Scholer, and Alistair Moffat. 2013. What Users Do: The Eyes Have It. In Asian Information Retrieval Symposium. 416--427.
[38]
Anastasios Tombros and Mark Sanderson. 1998. Advantages of Query Biased Summaries in Information Retrieval. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval. 2--10.
[39]
Michael Völske, Pavel Braslavski, Matthias Hagen, Galina Lezina, and Benno Stein. 2015. What Users Ask a Search Engine: Analyzing One Billion Russian Question Queries. 1571--1580.
[40]
Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating Vertical Results into Search Click Models. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 503--512.
[41]
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 189--198.
[42]
Allison Woodruff, Ruth Rosenholtz, Julie B. Morrison, Andrew Faulring, and Peter Pirolli. 2002. A Comparison of the Use of Text Summaries, Plain Thumbnails, and Enhanced Thumbnails for Web Search Tasks. Journal of the American Society for Information Science and Technology, Vol. 53, 2 (2002), 172--185.
[43]
Xiaohui Xie, Yiqun Liu, Xiaochuan Wang, Meng Wang, Zhijing Wu, Yingying Wu, Min Zhang, and Shaoping Ma. 2017. Investigating Examination Behavior of Image Search Users. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). 275--284.
[44]
Yukun Zheng, Jiaxin Mao, Yiqun Liu, Zixin Ye, Min Zhang, and Shaoping Ma. 2019. Human Behavior Inspired Machine Reading Comprehension. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Paris, France, 425--434.

Cited By

View all
  • (2024)Balancing Act: Boosting Strategies for Informed Search on Controversial TopicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638329(254-265)Online publication date: 10-Mar-2024
  • (2024)Robots still outnumber humans in web archives in 2019, but less than in 2015 and 2012International Journal on Digital Libraries10.1007/s00799-024-00397-225:3(537-553)Online publication date: 1-Sep-2024
  • (2023)How do Human and Contextual Factors Affect the Way People Formulate Queries?Proceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578336(499-503)Online publication date: 19-Mar-2023
  • Show More Cited By

Index Terms

  1. Providing Direct Answers in Search Results: A Study of User Behavior

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management
      October 2020
      3619 pages
      ISBN:9781450368599
      DOI:10.1145/3340531
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 October 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. answer module
      2. controlled user study
      3. eye-tracking
      4. question answering
      5. search behaviour
      6. user interaction
      7. web search

      Qualifiers

      • Research-article

      Funding Sources

      • Australian Research Council Project

      Conference

      CIKM '20
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

      Upcoming Conference

      CIKM '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)47
      • Downloads (Last 6 weeks)7
      Reflects downloads up to 01 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Balancing Act: Boosting Strategies for Informed Search on Controversial TopicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638329(254-265)Online publication date: 10-Mar-2024
      • (2024)Robots still outnumber humans in web archives in 2019, but less than in 2015 and 2012International Journal on Digital Libraries10.1007/s00799-024-00397-225:3(537-553)Online publication date: 1-Sep-2024
      • (2023)How do Human and Contextual Factors Affect the Way People Formulate Queries?Proceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578336(499-503)Online publication date: 19-Mar-2023
      • (2023)Investigating the Influence of Featured Snippets on User AttitudesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578323(211-220)Online publication date: 19-Mar-2023
      • (2023)The Evolution of Web Search User Interfaces - An Archaeological Analysis of Google Search Engine Result PagesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578320(55-68)Online publication date: 19-Mar-2023
      • (2023)Beyond Accurate Answers: Evaluating Open-Domain Question Answering in Enterprise SearchProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578314(308-312)Online publication date: 19-Mar-2023
      • (2023)In a Hurry: How Time Constraints and the Presentation of Web Search Results Affect User Behaviour and ExperienceWeb Engineering10.1007/978-3-031-34444-2_16(221-235)Online publication date: 6-Jun-2023
      • (2023)The influence of knowledge type and source reputation on preferences for website or video search resultsJournal of the Association for Information Science and Technology10.1002/asi.2477175:5(521-537)Online publication date: 17-May-2023
      • (2022)The Role of Information Boxes in Search Engine Results for Symptom Searches: Analysis of Archival DataJMIR Infodemiology10.2196/372862:2(e37286)Online publication date: 14-Sep-2022
      • (2022)Featured Snippets and their Influence on Users’ Credibility JudgementsProceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505766(113-122)Online publication date: 14-Mar-2022
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media