research-article

The good and the bad system: does the test collection predict users' effectiveness?

Authors:

Azzah Al-Maskari,

Mark Sanderson,

Paul Clough,

Eija AirioAuthors Info & Claims

SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

Pages 59 - 66

https://doi.org/10.1145/1390334.1390347

Published: 20 July 2008 Publication History

Get Access

Abstract

Test collections are extensively used in the evaluation of information retrieval systems. Crucial to their use is the degree to which results from them predict user effectiveness. At first, past studies did not substantiate a relationship between system and user effectiveness; more recently, however, correlations have begun to emerge. The results of this paper strengthen and extend those findings. We introduce a novel methodology for investigating the relationship, which shows great success in establishing a significant correlation between system and user effectiveness. It is shown that users behave differently and discern differences between pairs of systems that have a very small absolute difference in test collection effectiveness. Our results strengthen the use of test collections in IR evaluation, confirming that users' effectiveness can be predicted successfully.

References

[1]

J. Allan, J., Carterette, B. & Lewis, J. When Will Information Retrieval Be "Good Enough"? User Effectiveness As a Function of Retrieval Accuracy. ACM SIGIR, pages 433--440, Salvador, Brazil.2005

Digital Library

Google Scholar

[2]

Buckley, C. & E. M. Voorhees. Evaluating Evaluation Measure Stability. ACM SIGIR. Athens, Greece .2000

Digital Library

Google Scholar

[3]

Buckley, C. & E. M. Voorhees. Retrieval system Evaluation. In E. M. VOORHEES & HARMAN D. K. (Eds.), TREC: experiment and evaluation in information retrieval. London, England, MIT Press. 2005

Google Scholar

[4]

S. P. Harter, & C. A. Hert. Evaluation of information retrieval systems: Approaches, issues, and methods. Annual Review of Information Science and Technology (ARIST), 32, 3--94. 1997

Google Scholar

[5]

W. Hersh, A. Turpin, S. Price, B. Chan, D. Kraemer, L. Sacherek, and D. Olson. Do batch and user evaluations give the same results? In Proc. ACM SIGIR, pages 17--24, Athens, Greece, 2000.

Digital Library

Google Scholar

[6]

S. Huffman & M. Hochster. How Well does Result Relevance Predict Session Satisfaction?. ACM SIGIR.pages 567--573 Amsterdam, The Netherlands. 2007

Digital Library

Google Scholar

[7]

P. Ingwersen & K. Jäärvelin, The turn: integration of information seeking and retrieval in context, Springer. 2005

Digital Library

Google Scholar

[8]

T. Joachims, L. A. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proc. ACM SIGIR, pages 154--161, Salvador, Brazil, 2005.

Digital Library

Google Scholar

[9]

A. Turpin, & F. Scholer. User Performance versus Precision Measures for Simple Search Tasks. SIGIR, Pages 11--18. Seattle, Washington, USA. 2006

Digital Library

Google Scholar

[10]

A. Turpin, & W. Hersh. Why batch and user evaluations do not give the same results. ACM SIGIR, pages 225--231.New Orleans, Louisiana, United States. 2001

Digital Library

Google Scholar

[11]

A. Turpin, & W. Hersh. User interface effects in past batch versus user experiments. ACM SIGIR, pages 431--434. Tampere, Finland. 2002

Digital Library

Google Scholar

[12]

K. Yuri, & J. R. Moehr. Current Status of the Evaluation of Information Retrieval. Journal of Medical Systems 27(5): 409--424. 2003

Digital Library

Google Scholar

Cited By

View all

Rajagopal PAghris TFettah FRavana S(2023)Clustering of Relevant Documents Based on Findability Effort in Information RetrievalInternational Journal of Information Retrieval Research10.4018/IJIRR.31576412:1(1-18)Online publication date: 6-Jan-2023
https://doi.org/10.4018/IJIRR.315764
Wang XGao RJain AEdge GAhuja SChen HDuh WHuang HKato MMothe JPoblete B(2023)How Well do Offline Metrics Predict Online Performance of Product Ranking Models?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591865(3415-3420)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591865
Hajian Hoseinabadi ACheshmehSohrabi M(2022)Proposing a New Combined Indicator for Measuring Search Engine Performance and Evaluating Google, Yahoo, DuckDuckGo, and Bing Search Engines based on Combined IndicatorJournal of Librarianship and Information Science10.1177/0961000622113857956:1(178-197)Online publication date: 8-Dec-2022
https://doi.org/10.1177/09610006221138579
Show More Cited By

Index Terms

The good and the bad system: does the test collection predict users' effectiveness?
1. Information systems
  1. Information retrieval
  2. Information storage systems

Recommendations

NTCIR Lifelog: The First Test Collection for Lifelog Research
SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

Test collections have a long history of supporting repeatable and comparable evaluation in Information Retrieval (IR). However, thus far, no shared test collection exists for IR systems that are designed to index and retrieve multimodal lifelog data. In ...
Test collection management and labeling system
DocEng '09: Proceedings of the 9th ACM symposium on Document engineering

In order to evaluate the performance of information retrieval and extraction algorithms, we need test collections. A test collection consists of a set of documents, a clearly formed problem that an algorithm is supposed to provide solutions to, and the ...
Test collection recycling for semantic text similarity
IIWAS '12: Proceedings of the 14th International Conference on Information Integration and Web-based Applications & Services

Semantic text similarity (STS) uses specific test collections as its performance evaluation measurement. The test collections consist of text pairs with the same meaning even though in different text form. The existence is scarce compared with ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

July 2008

934 pages

ISBN:9781605581644

DOI:10.1145/1390334

General Chairs:
Tat-Seng Chua
National University of Singapore
,
Mun-Kew Leong
National Library Board, Singapore
,
Program Chairs:
Syung Hyon Myaeng
Information and Communications University, Korea
,
Douglas W. Oard
University of Maryland, College Park, USA
,
Fabrizio Sebastiani
Consiglio Nazionale delle Ricerche, Italy

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '08

Sponsor:

SIGIR '08: The 31st Annual International ACM SIGIR Conference

July 20 - 24, 2008

Singapore, Singapore

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
739
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rajagopal PAghris TFettah FRavana S(2023)Clustering of Relevant Documents Based on Findability Effort in Information RetrievalInternational Journal of Information Retrieval Research10.4018/IJIRR.31576412:1(1-18)Online publication date: 6-Jan-2023
https://doi.org/10.4018/IJIRR.315764
Wang XGao RJain AEdge GAhuja SChen HDuh WHuang HKato MMothe JPoblete B(2023)How Well do Offline Metrics Predict Online Performance of Product Ranking Models?Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591865(3415-3420)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591865
Hajian Hoseinabadi ACheshmehSohrabi M(2022)Proposing a New Combined Indicator for Measuring Search Engine Performance and Evaluating Google, Yahoo, DuckDuckGo, and Bing Search Engines based on Combined IndicatorJournal of Librarianship and Information Science10.1177/0961000622113857956:1(178-197)Online publication date: 8-Dec-2022
https://doi.org/10.1177/09610006221138579
Torfs-Leibman CAshikaga TKrag DLunna SRobtoy SBombardier R(2022)Pilot trial comparing COVID-19 publication database to conventional online search methodsBMJ Health & Care Informatics10.1136/bmjhci-2022-10061629:1(e100616)Online publication date: 8-Nov-2022
https://doi.org/10.1136/bmjhci-2022-100616
Jose KNguyen TMacAvaney SDalton JYates ADiaz FShah CSuel TCastells PJones RSakai T(2021)DiffIR: Exploring Differences in Ranking Models' BehaviorProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462784(2595-2599)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3462784
Sakai T(2021)On the Instability of Diminishing Return IR MeasuresAdvances in Information Retrieval10.1007/978-3-030-72113-8_38(572-586)Online publication date: 27-Mar-2021
https://doi.org/10.1007/978-3-030-72113-8_38
Alma’aitah WTalib AOsman M(2021)The Digital Resources Objects Retrieval: Concepts and FiguresInnovative Systems for Intelligent Health Informatics10.1007/978-3-030-70713-2_40(430-438)Online publication date: 6-May-2021
https://doi.org/10.1007/978-3-030-70713-2_40
Sakai TZeng Z(2020)Retrieval Evaluation Measures that Agree with Users’ SERP PreferencesACM Transactions on Information Systems10.1145/343181339:2(1-35)Online publication date: 31-Dec-2020
https://dl.acm.org/doi/10.1145/3431813
Sakai TZeng ZPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Which Diversity Evaluation Measures Are "Good"?Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331215(595-604)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331215
Voorhees E(2019)The Evolution of CranfieldInformation Retrieval Evaluation in a Changing World10.1007/978-3-030-22948-1_2(45-69)Online publication date: 14-Aug-2019
https://doi.org/10.1007/978-3-030-22948-1_2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

NTCIR Lifelog: The First Test Collection for Lifelog Research

Test collection management and labeling system

Test collection recycling for semantic text similarity

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations