Article

Free access

Do batch and user evaluations give the same results?

Authors:

Daniel OlsonAuthors Info & Claims

SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval

Pages 17 - 24

https://doi.org/10.1145/345508.345539

Published: 01 July 2000 Publication History

PDF eReader

Abstract

Do improvements in system performance demonstrated by batch evaluations confer the same benefit for real users? We carried out experiments designed to investigate this question. After identifying a weighting scheme that gave maximum improvement over the baseline in a non-interactive evaluation, we used it with real users searching on an instance recall task. Our results showed the weighting scheme giving beneficial results in batch studies did not do so with real users. Further analysis did identify other factors predictive of instance recall, including number of documents saved by the user, document recall, and number of documents seen by the user.

References

[1]

J. Chin, V. Diehl, and K. Norman, Development of an instrument measuring user satisfaction of the human-computer interface, Proceedings of CHI '88 - Human Factors in Computing Systems, New York, 213-218, 1988.

Digital Library

Google Scholar

[2]

C. Cleverdon and E. Keen, Factors determining the performance of indexin systems, Cranfield UK: Aslib Cranfield Research Project 1966.

Google Scholar

[3]

D. Harman, Overview of the first Text REtrieval Conference, Proceedings of the 16th Annual International ACM Special Interest Group in Information Retrieval, Pittsburgh, 36.47, 1993.

Digital Library

Google Scholar

[4]

W. Hersh, Relevance and retrieval evaluation: perspectives from medicine, Journal of the American Society for Information Science, 45: 201-206, 1994.

Digital Library

Google Scholar

[5]

E. Lagergren and P. Over, Comparing interactive information retrieval systems across sites: the TREC-6 interactive track matrix experiment, Proceedings of the 21st Annual International ACM SIGIR Conference on Research And Development in Information Retrieval, Melbroune, Australia, 162-172, 1998.

Digital Library

Google Scholar

[6]

C. Meadow, Relevance?, Journal of the American Society for Information Science, 36: 354-355, 1985.

Crossref

Google Scholar

[7]

S. Robertson and S. Walker, Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval, Proceedings of the 17th Annual International ACM Special Interest Group in Information Retrieval, Dublin, 232-241, 1994.

Digital Library

Google Scholar

[8]

A. Singhal, C. Buckley, and M. Mitra, Pivoted document length normalization, Proceedings of the 19th Annual International A CM Special Interest Group in Information Retrieval, Zurich, Switzerland, 21-29, 1996.

Digital Library

Google Scholar

[9]

K. Sparck-Jones, Information Retrieval Experiment. London: Butterworths, 1981.

Digital Library

Google Scholar

[10]

D. Swanson, Information retrieval as a trial-anderror process, Library Quarterly, 47: 128-148, 1977.

Crossref

Google Scholar

[11]

I. Witten, A. Moffat, and T. Bell, Managing Gigabytes - Compressing and lndexing Documents and Images. New York: Van Nostrand Reinhold, 1994.

Digital Library

Google Scholar

[12]

J. Zobel and A. Moffat, Exploring the similarity space, SIGIR Forum, 32: 18-34, 1998.

Digital Library

Google Scholar

Cited By

View all

Balog KZhai CChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Tutorial on User Simulation for Evaluating Information Access Systems on the WebCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3641243(1254-1257)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641243
Engelmann BBreuer TFriese JSchaer PFuhr N(2024)Context-Driven Interactive Query Simulations Based on Generative Large Language ModelsAdvances in Information Retrieval10.1007/978-3-031-56060-6_12(173-188)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56060-6_12
Balog KZhai C(2023)User Simulation for Evaluating Information Access SystemsProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3629549(302-305)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3629549
Show More Cited By

Index Terms

Do batch and user evaluations give the same results?
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

Why batch and user evaluations do not give the same results
SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval

Much system-oriented evaluation of information retrieval systems has used the Cranfield approach based upon queries run against test collections in a batch mode. Some researchers have questioned whether this approach can be applied to the real world, ...
Batch Public Key Cryptosystem with batch multi-exponentiation

A Public Key Cryptosystem (PKC) is a fundamental tool to protect data security. Most PKC schemes involve complicated operations, e.g., modular exponentiations, which are expensive for cloud environment where enormous data are collected from capability-...
User-tailored web accessibility evaluations
HT '07: Proceedings of the eighteenth conference on Hypertext and hypermedia

This paper presents a framework and system to evaluate the accessibility of web pages according to the individual requirements of users with disabilities. These requirements not only consist of users' abilities, but also users' assistive technologies ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval

July 2000

396 pages

ISBN:1581132263

DOI:10.1145/345508

Chairmen:
Emmanuel Yannakoudakis
Athens Univ. of Economics and Business, Greece
,
Nicholas J. Belkin
Rutgers Univ.
,
Mun-Kew Leong
Kent Ridge Digital Labs
,
Peter Ingwersen
Royal School of Library and Information Science

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2000

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGIR00

Sponsor:

Greek Com Soc
SIGIR
Athens U of Econ & Business

SIGIR00: 23rd ACM International SIGIR Conference on Research and Development in Information Retrieval

July 24 - 28, 2000

Athens, Greece

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

89
Total Citations
View Citations
186
Total Downloads

Downloads (Last 12 months)45
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Balog KZhai CChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Tutorial on User Simulation for Evaluating Information Access Systems on the WebCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3641243(1254-1257)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641243
Engelmann BBreuer TFriese JSchaer PFuhr N(2024)Context-Driven Interactive Query Simulations Based on Generative Large Language ModelsAdvances in Information Retrieval10.1007/978-3-031-56060-6_12(173-188)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56060-6_12
Balog KZhai C(2023)User Simulation for Evaluating Information Access SystemsProceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3624918.3629549(302-305)Online publication date: 26-Nov-2023
https://dl.acm.org/doi/10.1145/3624918.3629549
Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Balog KZhai CFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Tutorial on User Simulation for Evaluating Information Access SystemsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615296(5200-5203)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615296
Baghestani Tajali ASanatjoo ABehzadi HJamali H(2021)Use of mind mapping in search process to clarify information needs and improve search satisfactionJournal of Information Science10.1177/0165551521105804149:5(1417-1427)Online publication date: 13-Dec-2021
https://doi.org/10.1177/01655515211058041
Lima de Oliveira LRomeu RMoreira VDiaz FShah CSuel TCastells PJones RSakai T(2021)REGIS: A Test Collection for Geoscientific Documents in PortugueseProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463256(2363-2368)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463256
van der Vegt AZuccon GKoopman B(2021)Do better search engines really equate to better clinical decisions? If not, why not?Journal of the Association for Information Science and Technology10.1002/asi.2439872:2(141-155)Online publication date: 18-Jan-2021
https://dl.acm.org/doi/10.1002/asi.24398
Sakai TZeng ZHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Good Evaluation Measures based on Document PreferencesProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401115(359-368)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401115
Hersh WHersh W(2020)ResearchInformation Retrieval: A Biomedical and Health Perspective10.1007/978-3-030-47686-1_8(337-405)Online publication date: 23-Jul-2020
https://doi.org/10.1007/978-3-030-47686-1_8
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Why batch and user evaluations do not give the same results

Batch Public Key Cryptosystem with batch multi-exponentiation

User-tailored web accessibility evaluations