Article

Test theory for assessing IR test collections

Authors:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 367 - 374

https://doi.org/10.1145/1277741.1277805

Published: 23 July 2007 Publication History

Get Access

Abstract

How good is an IR test collection? A series of papers in recent years has addressed the question by empirically enumerating the consistency of performance comparisons using alternate subsets of the collection. In this paper we propose using Test Theory, which is based on analysis of variance and is specifically designed to assess test collections. Using the method, we not only can measure test reliability after the fact, but we can estimate the test collection's reliability before it is even built or used. We can also determine an optimal allocation of resources before the fact, e.g. whether to invest in more judges or queries. The method, which is in widespread use in the field of educational testing, complements data-driven approaches to assessing test collections. Whereas the data-driven method focuses on test results, test theory focuses on test designs. It offers unique practical results, as well as insights about the variety and implications of alternative test designs.

References

[1]

Zobel, J., How reliable are the results of large-scale information retrieval experiments?, in Proceedings of the 21st ACM SIGIR Conference on Research and Development In Information Retrieval. 1998, ACM: Melbourne.

Digital Library

Google Scholar

[2]

Voorhees, E. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. in Proceedings of the 21st ACM SIGIR Conference on Research and Development In Information Retrieval. 1998. Melbourne: ACM Press.

Digital Library

Google Scholar

[3]

Buckley, C. and E. Voorhees. Evaluating Evaluation Measure Stability. in Proceedings of the 23rd ACM SIGIR conference on Research and Development in Information Retrieval. 2000. Athens, Greece: ACM Press.

Digital Library

Google Scholar

[4]

Voorhees, E.M. and C. Buckley. The Effect of Topic Set Size on Retrieval Experiment Error. in Proceedings of the 25th ACM SIGIR Conference on Research and Development In Information Retrieval. 2002. Tampere: ACM Press.

Digital Library

Google Scholar

[5]

Sanderson, M. and J. Zobel, Information retrieval system evaluation: effort, sensitivity, and reliability, in Proceedings of the 28th ACM SIGIR Conference on Research and Development in Information Retrieval. 2005, ACM: Salvador, Brazil.

Digital Library

Google Scholar

[6]

Banks, D., P. Over, and N.F. Zhang, Blind men and elephants: six approaches to TREC data. Information Retrieval, 1999. 1(1-2): p. 7--34.

Digital Library

Google Scholar

[7]

Lange, R., et al., A probabilistic Rasch analysis of question answering evaluation, in Proceedings of Human Language Technology conference North American chapter of the Association for Computational Linguistics annual meeting, HLT/NAACL 2004. 2004, Association for Computational Linguistics: Boston.

Google Scholar

[8]

Crocker, L. and J. Algina, Introduction to Classical & Modern Test Theory. 1986: Holt, Rinehart, and Winston.

Google Scholar

[9]

Brennan, R.L., Generalizability Theory. Statistics for Social Science and Public Policy, ed. S.E.D. Lievesley and J.R. Feinberg. 2001: Springer-Verlag.

Google Scholar

[10]

Shavelson, R.J. and N.M. Webb, Generalizability Theory: A Primer. 1991, Newbury Park, CA: Sage.

Google Scholar

[11]

Gao, X. and L. Brennan, Variability of Estimated Variance Components and Related Statistics in a Performance Assessment. Applied Measurement in Education, 2001. 14(2): p. 191--203.

Crossref

Google Scholar

Cited By

View all

Faggioli GFerro NFuhr NAl Hasan MXiong L(2022)Detecting Significant Differences Between Information Retrieval Systems via Generalized Linear ModelsProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557286(446-456)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557286
Carmel DCohen NIngber AKravi EAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)IR Evaluation and Learning in the Presence of Forbidden DocumentsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532006(556-566)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532006
Culpepper JFaggioli GFerro NKurland O(2021)Topic Difficulty: Collection and Query Formulation EffectsACM Transactions on Information Systems10.1145/347056340:1(1-36)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3470563
Show More Cited By

Index Terms

Test theory for assessing IR test collections
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
  2. Information storage systems

Recommendations

Test theory for evaluating reliability of IR test collections

Classical test theory offers theoretically derived reliability measures such as Cronbach's alpha, which can be applied to measure the reliability of a set of Information Retrieval test results. The theory also supports item analysis, which identifies ...
Dynamic Test Collections for Retrieval Evaluation
ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

Batch evaluation with test collections of documents, search topics, and relevance judgments has been the bedrock of IR evaluation since its adoption by Salton for his experiments on vector space systems. Such test collections have limitations: they ...
Reusable test collections through experimental design
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval

Portable, reusable test collections are a vital part of research and development in information retrieval. Reusability is difficult to assess, however. The standard approach--simulating judgment collection when groups of systems are held out, then ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
703
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)2

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Faggioli GFerro NFuhr NAl Hasan MXiong L(2022)Detecting Significant Differences Between Information Retrieval Systems via Generalized Linear ModelsProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557286(446-456)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557286
Carmel DCohen NIngber AKravi EAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)IR Evaluation and Learning in the Presence of Forbidden DocumentsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532006(556-566)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532006
Culpepper JFaggioli GFerro NKurland O(2021)Topic Difficulty: Collection and Query Formulation EffectsACM Transactions on Information Systems10.1145/347056340:1(1-36)Online publication date: 8-Sep-2021
https://dl.acm.org/doi/10.1145/3470563
Faggioli GFerro N(2021)System Effect Estimation by Sharding: A Comparison Between ANOVA Approaches to Detect Significant DifferencesAdvances in Information Retrieval10.1007/978-3-030-72240-1_3(33-46)Online publication date: 30-Mar-2021
https://doi.org/10.1007/978-3-030-72240-1_3
Gienapp LFröbe MHagen MPotthast Md'Aquin MDietze SHauff CCurry ECudre Mauroux P(2020)The Impact of Negative Relevance Judgments on NDCGProceedings of the 29th ACM International Conference on Information & Knowledge Management10.1145/3340531.3412123(2037-2040)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3340531.3412123
Ferro NSanderson MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Improving the Accuracy of System Performance Estimation by Using ShardsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3338062(805-814)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3338062
Ferro NKim YSanderson M(2019)Using Collection Shards to Study Retrieval Performance Effect SizesACM Transactions on Information Systems10.1145/331036437:3(1-40)Online publication date: 19-Mar-2019
https://dl.acm.org/doi/10.1145/3310364
Yaghtin MSotudeh HMirzabeigi MFakhrahmad SMohammadi M(2019)In quest of new document relationsScientometrics10.1007/s11192-019-03058-3119:2(987-1008)Online publication date: 1-May-2019
https://dl.acm.org/doi/10.1007/s11192-019-03058-3
Roitero KCulpepper JSanderson MScholer FMizzaro S(2019)Fewer topics? A million topics? Both?! On topics subsets in test collectionsInformation Retrieval Journal10.1007/s10791-019-09357-wOnline publication date: 8-May-2019
https://doi.org/10.1007/s10791-019-09357-w
Losada DParapar JBarreiro A(2018)When to stop making relevance judgments? A study of stopping methods for building information retrieval test collectionsJournal of the Association for Information Science and Technology10.1002/asi.2407770:1(49-60)Online publication date: 12-Dec-2018
https://dl.acm.org/doi/10.1002/asi.24077
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Test theory for evaluating reliability of IR test collections

Dynamic Test Collections for Retrieval Evaluation

Reusable test collections through experimental design