More Web Proxy on the site http://driver.im/

Article

Retrieval evaluation with incomplete information

Authors:

Ellen M. VoorheesAuthors Info & Claims

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 25 - 32

https://doi.org/10.1145/1008992.1009000

Published: 25 July 2004 Publication History

Abstract

This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the collection). We show that current evaluation measures are not robust to substantially incomplete relevance judgments. A new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets. This finding suggests that substantially larger or dynamic test collections built using current pooling practices should be viable laboratory tools, despite the fact that the relevance information will be incomplete and imperfect.

References

[1]

Chris Buckley. trec_eval IR evaluation package. Available from ftp://ftp.cs.cornell.edu/pub/smart.

[2]

Chris Buckley and EllenM. Voorhees. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 33--40, 2000.

Digital Library

[3]

Cyril W. Cleverdon. The significance of the Cranfield tests on index languages. In Proceedings of the Fourteenth Annual International ACM/SIGIR Conference on Reserach and Development in Information Retrieval, pages 3--12, 1991.

Digital Library

[4]

Gordon V. Cormack, ChristopherR. Palmer, and CharlesL.A. Clarke. Efficient construction of large test collections. In Croft et al. sigir98, pages 282--289.

Digital Library

[5]

W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, August 1998. ACM Press, New York.

[6]

M. B. Eisenberg. Measuring relevance judgments. Information Processing and Management, 24(4):373--389, 1988.

Digital Library

[7]

H. P. Frei and P. Schäuble. Determining the effectiveness of retrieval algorithms. Information Processing and Management, 27(2/3):153--164, 1991.

Digital Library

[8]

Google. Benefits of a Google search. http://www.google.com/technology/whyuse.html, January 2004.

[9]

Stefano Mizzaro. A new measure of retrieval effectiveness(Or: What's wrong with precision and recall). In Proceedings of the International Workshop on Information Retrieval(IR'2001), pages 43--52, 2001.

[10]

Rabia Nuray and Fazli Can. Automatic ranking of retrieval systems in imperfect environments. In Proceedings of the Twenty-Sixth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(SIGIR 2003), pages 379--380, 2003.

Digital Library

[11]

Mark E. Rorvig. The simple scalability of documents. Journal of the American Society for Information Science, 41(8):590--598, 1990.

[12]

Ian Soboroff, Charles Nicholas, and Patrick Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66--73, 2001.

Digital Library

[13]

K. Sparck Jones and C. van Rijsbergen. Report on the need for and provision of an "ideal" information retrieval test collection. British Library Research and Development Report 5266, Computer Laboratory, University of Cambridge, 1975.

[14]

C. J. van Rijsbergen. Evaluation, chapter 7. Butterworths, 2 edition, 1979.

[15]

Ellen M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing and Management, 36:697--716, 2000.

Digital Library

[16]

Ellen M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74--82, 2001.

Digital Library

[17]

Ellen M. Voorhees. The philosophy of information retrieval evaluation. In Evaluation of Cross-Language Information Retrieval Systems. Proceedings of CLEF 2001, number 2406 in Lecture Notes in Computer Science, pages 355--370, 2002.

Digital Library

[18]

Ellen M. Voorhees and Chris Buckley. The effect of topic set size on retrieval experiment error. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 316--323, 2002.

Digital Library

[19]

Ellen M. Voorhees and Donna Harman. Overview of the seventh Text REtrieval Conference(TREC-7). In Proceedings of the Seventh Text REtrieval Conference(TREC-7), pages 1--23, 1999. NIST Special Publication 500--242.

[20]

Y. Y. Yao. Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information Science, 46(2):133--145, 1995.

Digital Library

[21]

Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Croft et al. {5}, pages 307--314.

Digital Library

Cited By

Dadure PPakray PBandyopadhyay S(2024)Mathematical Information Retrieval: A ReviewACM Computing Surveys10.1145/369995357:3(1-34)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3699953
Keller JBreuer TSchaer POosterhuis HBast HXiong C(2024)Evaluation of Temporal Change in IR Test CollectionsProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672530(3-13)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672530
Thomas PKazai GCraswell NSpielman SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)What Matters in a Measure? A Perspective from Large-Scale Search EvaluationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657845(282-292)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657845
Show More Cited By

Index Terms

Retrieval evaluation with incomplete information
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

Evaluation of retrieval effectiveness with incomplete relevance data: Theoretical and experimental comparison of three measures

This paper investigates two relatively new measures of retrieval effectiveness in relation to the problem of incomplete relevance data. The measures, Bpref and RankEff, which do not take into account documents that have not been relevance judged, are ...
Estimating average precision when judgments are incomplete

We consider the problem of evaluating retrieval systems with incomplete relevance judgments. Recently, Buckley and Voorhees showed that standard measures of retrieval performance are not robust to incomplete judgments, and they proposed a new measure, ...
On the robustness of relevance measures with incomplete judgments
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

We investigate the robustness of three widely used IR relevance measures for large data collections with incomplete judgments. The relevance measures we consider are the bpref measure introduced by Buckley and Voorhees [7], the inferred average ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

July 2004

624 pages

ISBN:1581138814

DOI:10.1145/1008992

General Chair:
Mark Sanderson
University of Sheffield (UK)
,
Program Chairs:
Kalervo Järvelin
University of Tampere (Finland)
,
James Allan
University of Massachusetts (USA)
,
Peter Bruza
Distributed Systems Technology Centre (Australia)

Copyright © 2004 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR04

Sponsor:

SIGIR04: The 27th ACM/SIGIR International Symposium on Information Retrieval 2004

July 25 - 29, 2004

Sheffield, United Kingdom

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

435
Total Citations
View Citations
3,253
Total Downloads

Downloads (Last 12 months)90
Downloads (Last 6 weeks)15

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dadure PPakray PBandyopadhyay S(2024)Mathematical Information Retrieval: A ReviewACM Computing Surveys10.1145/369995357:3(1-34)Online publication date: 9-Oct-2024
https://dl.acm.org/doi/10.1145/3699953
Keller JBreuer TSchaer POosterhuis HBast HXiong C(2024)Evaluation of Temporal Change in IR Test CollectionsProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672530(3-13)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672530
Thomas PKazai GCraswell NSpielman SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)What Matters in a Measure? A Perspective from Large-Scale Search EvaluationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657845(282-292)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657845
Jaiswal AKumar MPathak AYigzaw K(2024)FaN-REMs: Fair and Normalized Retrieval Evaluation Metrics for Learning Retrieval SystemsIEEE Access10.1109/ACCESS.2024.351491612(195370-195395)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3514916
Raju KReddy NPraveen MChenna VSandhyarani Chandana B(2024)Sentiment Analysis Principle Technical Approach on Online Social Network Data Using CNN for Detection of StressProceedings of Fifth International Conference on Computer and Communication Technologies10.1007/978-981-99-9704-6_37(401-410)Online publication date: 14-Feb-2024
https://doi.org/10.1007/978-981-99-9704-6_37
Chen YWan JDu YYu JHou LLi J(2024)Entity Set Expansion Based on Category Prompts in MOOCsKnowledge Science, Engineering and Management10.1007/978-981-97-5495-3_24(318-332)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-981-97-5495-3_24
Schultzberg COttens B(2024)Navigating the Evaluation Funnel to Optimize Iteration Speed for Recommender SystemsProceedings of the Future Technologies Conference (FTC) 2024, Volume 110.1007/978-3-031-73110-5_11(138-157)Online publication date: 5-Nov-2024
https://doi.org/10.1007/978-3-031-73110-5_11
Keller JBreuer TSchaer P(2024)Replicability Measures for Longitudinal Information Retrieval EvaluationExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-71736-9_16(215-226)Online publication date: 14-Sep-2024
https://doi.org/10.1007/978-3-031-71736-9_16
Wang PChen H(2024)The Effectiveness of Graph Contrastive Learning on Mathematical Information RetrievalAdvances on Graph-Based Approaches in Information Retrieval10.1007/978-3-031-71382-8_5(60-72)Online publication date: 10-Oct-2024
https://doi.org/10.1007/978-3-031-71382-8_5
Giner F(2024)An Intrinsic Framework of Information Retrieval Evaluation MeasuresIntelligent Systems and Applications10.1007/978-3-031-47721-8_47(692-713)Online publication date: 10-Jan-2024
https://doi.org/10.1007/978-3-031-47721-8_47
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten