short-paper

Unbiased Low-Variance Estimators for Precision and Related Information Retrieval Effectiveness Measures

Authors:

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 945 - 948

https://doi.org/10.1145/3331184.3331355

Published: 18 July 2019 Publication History

Get Access

Abstract

This work describes an estimator from which unbiased measurements of precision, rank-biased precision, and cumulative gain may be derived from a uniform or non-uniform sample of relevance assessments. Adversarial testing supports the theory that our estimator yields unbiased low-variance measurements from sparse samples, even when used to measure results that are qualitatively different from those returned by known information retrieval methods. Our results suggest that test collections using sampling to select documents for relevance assessment yield more accurate measurements than test collections using pooling, especially for the results of retrieval methods not contributing to the pool.

References

[1]

Aslam, J. A., Pavlu, V., and Savell, R. A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm. In SIGIR 2003.

Digital Library

Google Scholar

[2]

Cormack, G. V., and Grossman, M. R. Beyond pooling. In SIGIR 2018.

Digital Library

Google Scholar

[3]

Horvitz, D. G., and Thompson, D. J. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association 47, 260 (1952), 663--685.

Crossref

Google Scholar

[4]

Pavlu, V., and Aslam, J. A practical sampling strategy for efficient retrieval evaluation. Northeastern University (2007).

Google Scholar

[5]

Sanderson, M., et al. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval 4, 4 (2010), 247--375.

Crossref

Google Scholar

[6]

Voorhees, E., and Harman, D. Overview of the eighth text retrieval conference. In TREC 8 (1999).

Google Scholar

[7]

Voorhees, E. M. The effect of sampling strategy on inferred measures. In SIGIR 2014.

Digital Library

Google Scholar

[8]

Yilmaz, E., Kanoulas, E., and Aslam, J. A. A simple and efficient sampling method for estimating AP and NDCG. In SIGIR 2008.

Digital Library

Google Scholar

Cited By

View all

Cormack GGrossman MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Quantifying Bias and Variance of System RankingsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331356(1089-1092)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331356
Cormack GZhang HGhelani NAbualsaud MSmucker MGrossman MRahbariasl SGhenai APiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Dynamic Sampling Meets PoolingProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331354(1217-1220)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331354

Index Terms

Unbiased Low-Variance Estimators for Precision and Related Information Retrieval Effectiveness Measures
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Retrieval effectiveness
      2. Test collections

Recommendations

Beyond Pooling
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Dynamic Sampling is a novel, non-uniform, statistical sampling strategy in which documents are selected for relevance assessment based on the results of prior assessments. Unlike static and dynamic pooling methods that are commonly used to compile ...
Ratio estimators for the population variance in simple and stratified random sampling

We propose some ratio-type variance estimators using ratio estimators for the population mean in literature. We obtain mean square error (MSE) equations of proposed estimators and show that proposed estimators are more efficient than the traditional ...
Assessing the Impact of Vocabulary Similarity on Multilingual Information Retrieval for Bantu Languages
FIRE '16: Proceedings of the 8th Annual Meeting of the Forum for Information Retrieval Evaluation

Despite the availability of massive open information and efforts to promote multilingualism on the Web, content in Bantu languages remains negligible. Additionally, Information Retrieval (IR) systems, such as the Google search engine, use algorithms ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2019

1512 pages

ISBN:9781450361729

DOI:10.1145/3331184

General Chairs:
Benjamin Piwowarski
CNRS - Sorbonne Universite, France
,
Max Chevalier
Universite de Toulouse, CNRS, France
,
Eric Gaussier
Universite Grenoble Alpes, CNRS, France
,
Program Chairs:
Yoelle Maarek
Amazon Research, Israel
,
Jian-Yun Nie
University of Montreal, Canada
,
Falk Scholer
RMIT University, Australia

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2019

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '19

Sponsor:

SIGIR

SIGIR '19: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 21 - 25, 2019

Paris, France

Acceptance Rates

SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
208
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Cormack GGrossman MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Quantifying Bias and Variance of System RankingsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331356(1089-1092)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331356
Cormack GZhang HGhelani NAbualsaud MSmucker MGrossman MRahbariasl SGhenai APiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Dynamic Sampling Meets PoolingProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331354(1217-1220)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331354

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Beyond Pooling

Ratio estimators for the population variance in simple and stratified random sampling

Assessing the Impact of Vocabulary Similarity on Multilingual Information Retrieval for Bantu Languages