More Web Proxy on the site http://driver.im/

short-paper

Detecting cognitive bias in a relevance assessment task using an eye tracker

Author:

Christopher G. HarrisAuthors Info & Claims

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Article No.: 36, Pages 1 - 5

https://doi.org/10.1145/3314111.3319824

Published: 25 June 2019 Publication History

Abstract

Cognitive biases, such as the bandwagon effect, occur when a participant places a disproportionate emphasis on external information when making decisions under uncertainty. These effects are challenging for humans to overcome - even when they are explicitly made aware of their own biases. One challenge for researchers is to detect if the information is used in decision making and to what degree. One can gain a better understanding of how this external information is used in decision making using an eye tracker. In this paper, we evaluate cognitive biases in the context of assessing the binary relevance of a set of documents in response to a given information need. We show that these cognitive biases can be observed by examining gaze time in Areas of Interest (AOI) that contain this pertinent external information.

References

[1]

Nilavra Bhattacharya & Jacek Gwizdka, 2018. Relating eye-tracking measures with changes in knowledge on search tasks. In Proceedings of the. 2018 ACM Symposium on Eye Tracking Research & Applications Article 62. ACM.

Digital Library

[2]

Carsten Eickhoff. 2018. Cognitive Biases in Crowdsourcing. Proc. 11th ACM International Conference on Web Search and Data Mining, pp. 162--170. ACM.

Digital Library

[3]

Robert Epstein, Robertson, R. E., Lazer, D., & Wilson, C. (2017). Suppressing the search engine manipulation effect (SEME). In Proceedings of the ACM on Human-Computer Interaction, 1(CSCW), 42.

Digital Library

[4]

Jacek Gwizdka, 2014. Characterizing relevance with eye-tracking measures. In Proceedings of the 5th Information Interaction in Context Symposium (pp. 58--67). ACM.

Digital Library

[5]

Martie G. Haselton, D. Nettle, and P.W. Andrews. 2005. The evolution of cognitive bias. In D. M. Buss (Ed.), The Handbook of Evolutionary Psychology: Hoboken, NJ, US: John Wiley & Sons Inc. pp. 724--746.

[6]

William Hersh, W., Cohen, A., et al. 2007. TREC 2007 genomics track overview. The Sixteenth Text Retrieval Conference (TREC 2007). Gaithersburg, MD: National Institute for Standards and Technology.

[7]

Kirsten Kirkegaard Moe, Jensen, J. M., & Larsen, B. 2007. A qualitative look at eye-tracking for implicit relevance feedback. In Proceedings of the Workshop on Context-Based Information Retrieval (Vol. 326, pp. 36--47).

[8]

Bing Pan, Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. 2007. In google we trust: Users' decisions on rank, position, and relevance. Journal of computer-mediated communication, 12(3), 801--823.

[9]

Mark Sanderson and W.B. Croft. 2012. The history of information retrieval research. Proc. IEEE 100, no. Special Centennial Issue pp. 1444--1451.

[10]

Falk Scholer, D. Kelly, W.C. Wu, H. S. Lee, and W. Webber. 2013. The effect of threshold priming and need for cognition on relevance calibration and assessment. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval., pp. 623--632. ACM.

Digital Library

[11]

Milad Shokouhi, Milad, R. White, and E. Yilmaz. 2015. Anchoring and adjustment in relevance estimation. In Proceedings of the 38th International ACM SIGIR Conference on research and development in information retrieval., pp. 963--966. ACM.

Digital Library

[12]

Christop Schneider, Weinmann, M., & vom Brocke, J. 2015. Choice architecture: Using fixation patterns to analyze the effects of form design on cognitive biases. Information Systems and Neuroscience (pp. 91--97). Springer, Cham.

[13]

Amos Tversky and D. Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157): pp. 1124--1131.

[14]

Xuanhui Wang, Bendersky, M., Metzler, D., & Najork, M. 2016. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (pp. 115--124). ACM.

Digital Library

[15]

Ryen White. 2013. Beliefs and biases in web search. In Proc. 36th Int'l ACM SIGIR conference on Research and development in information retrieval (pp. 3--12). ACM

Digital Library

[16]

Ryen W. White & Hassan, A. 2014. Content bias in online health search. ACM Transactions on the Web (TWEB), 8(4), 25.

Digital Library

[17]

Ryen W. White, & Horvitz, E. 2015. Belief dynamics and biases in web search. ACM Transactions on Information Systems (TOIS), 33(4), 18.

Digital Library

Cited By

Azzopardi LLiu JSakai TIshita EOhshima HRadboud University, Netherlands FMao JJose J(2024)Evaluating Cognitive Biases in Conversational and Generative IIR: A TutorialProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698437(287-290)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698437
Ji KPathiyan Cherumanal STrippas JHettiachchi DSalim FScholer FSpina D(2024)Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational SearchAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680245(1-10)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680245
Azzopardi LLiu J(2024)Search under Uncertainty: Cognitive Biases and Heuristics - Tutorial on Modeling Search Interaction using Behavioral EconomicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638297(427-430)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638297
Show More Cited By

Index Terms

Detecting cognitive bias in a relevance assessment task using an eye tracker
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment

Recommendations

Task-embedded online eye-tracker calibration for improving robustness to head motion
ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

Remote eye trackers are widely used for screen-based interactions. They are less intrusive than head mounted eye trackers, but are generally quite sensitive to head movement. This leads to the requirement for frequent recalibration, especially in ...
Measuring the task-evoked pupillary response with a remote eye tracker
ETRA '08: Proceedings of the 2008 symposium on Eye tracking research & applications

The pupil-measuring capability of video eye trackers can detect the task-evoked pupillary response: subtle changes in pupil size which indicate cognitive load. We performed several experiments to measure cognitive load using a remote video eye tracker, ...
Eye detection and coarse localization of pupil for video-based eye tracking systems
Abstract
A video-based eye tracking system generally captures NIR images, each of which contains one or two eyes of a subject. The subject’s point of gaze is then determined using 3D eye model and pupil centre corneal reflection technique. Eye detection ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '19: Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications

June 2019

623 pages

ISBN:9781450367097

DOI:10.1145/3314111

Conference Chairs:
Krzysztof Krejtz
SWPS University, Poland
,
Bonita Sharif
University of Nebraska-Lincoln

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ETRA '19

Sponsor:

ETRA '19: 2019 Symposium on Eye Tracking Research and Applications

June 25 - 28, 2019

Colorado, Denver

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
315
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)12

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Azzopardi LLiu JSakai TIshita EOhshima HRadboud University, Netherlands FMao JJose J(2024)Evaluating Cognitive Biases in Conversational and Generative IIR: A TutorialProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698437(287-290)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698437
Ji KPathiyan Cherumanal STrippas JHettiachchi DSalim FScholer FSpina D(2024)Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational SearchAdjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction10.1145/3640471.3680245(1-10)Online publication date: 21-Sep-2024
https://dl.acm.org/doi/10.1145/3640471.3680245
Azzopardi LLiu J(2024)Search under Uncertainty: Cognitive Biases and Heuristics - Tutorial on Modeling Search Interaction using Behavioral EconomicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638297(427-430)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638297
Liu JAzzopardi LHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Search under Uncertainty: Cognitive Biases and Heuristics: A Tutorial on Testing, Mitigating and Accounting for Cognitive Biases in Search ExperimentsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661382(3013-3016)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661382
Albaghli RGurnaney AAttar N(2024)Detecting Bias in Refugee Perception using Face Swapping: An Empirical Eye-Tracking Study2024 IEEE 4th International Conference on Human-Machine Systems (ICHMS)10.1109/ICHMS59971.2024.10555693(1-6)Online publication date: 15-May-2024
https://doi.org/10.1109/ICHMS59971.2024.10555693
Boonprakong NHe GGadiraju Uvan Berkel NWang DChen SLiu JTag BGoncalves JDingler T(2023)Workshop on Understanding and Mitigating Cognitive Biases in Human-AI CollaborationCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3611284(512-517)Online publication date: 14-Oct-2023
https://dl.acm.org/doi/10.1145/3584931.3611284
Saleem MMayne RNapolitano R(2023)Analysis of gaze patterns during facade inspection to understand inspector sense-making processesScientific Reports10.1038/s41598-023-29950-w13:1Online publication date: 20-Feb-2023
https://doi.org/10.1038/s41598-023-29950-w
Mantri PSubramonyam HMichal AXiong C(2022)How Do Viewers Synthesize Conflicting Information from Data Visualizations?IEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.3209467(1-11)Online publication date: 2022
https://doi.org/10.1109/TVCG.2022.3209467
Geng BVarshney P(2022)Human-Machine Collaboration for Smart Decision Making: Current Trends and Future Opportunities2022 IEEE 8th International Conference on Collaboration and Internet Computing (CIC)10.1109/CIC56439.2022.00019(61-67)Online publication date: Dec-2022
https://doi.org/10.1109/CIC56439.2022.00019
Azzopardi LScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Cognitive Biases in SearchProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446023(27-37)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446023

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents