Export Citations
Save this search
Please login to be able to save your searches and receive alerts for new content matching your search criteria.
- research-articleJuly 2022
Too Many Relevants: Whither Cranfield Test Collections?
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information RetrievalPages 2970–2980https://doi.org/10.1145/3477495.3531728This paper presents the lessons regarding the construction and use of large Cranfield-style test collections learned from the TREC 2021 Deep Learning track. The corpus used in the 2021 edition of the track was much bigger than the corpus used previously ...
- research-articleNovember 2017
Active Sampling for Large-scale Information Retrieval Evaluation
CIKM '17: Proceedings of the 2017 ACM on Conference on Information and Knowledge ManagementPages 49–58https://doi.org/10.1145/3132847.3133015Evaluation is crucial in Information Retrieval. The development of models, tools and methods has significantly benefited from the availability of reusable test collections formed through a standardized and thoroughly tested methodology, known as the ...
- research-articleJuly 2010
Human performance and retrieval precision revisited
SIGIR '10: Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrievalPages 595–602https://doi.org/10.1145/1835449.1835549Several studies have found that the Cranfield approach to evaluation can report significant performance differences between retrieval systems for which little to no performance difference is found for humans completing tasks with these systems. We ...
- ArticleJuly 2004
Retrieval evaluation with incomplete information
SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrievalPages 25–32https://doi.org/10.1145/1008992.1009000This paper examines whether the Cranfield evaluation methodology is robust to gross violations of the completeness assumption (i.e., the assumption that all relevant documents within a test collection have been identified and are present in the ...