Article

A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm

Authors:

SIGIR '03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval

Pages 393 - 394

https://doi.org/10.1145/860435.860517

Published: 28 July 2003 Publication History

Get Access

Abstract

We present a unified framework for simultaneously solving both the pooling problem (the construction of efficient document pools for the evaluation of retrieval systems) and metasearch (the fusion of ranked lists returned by retrieval systems in order to increase performance). The implementation is based on the Hedge algorithm for online learning, which has the advantage of convergence to bounded error rates approaching the performance of the best linear combination of the underlying systems. The choice of a loss function closely related to the average precision measure of system performance ensures that the judged document set performs well, both in constructing a metasearch list and as a pool for the accurate evaluation of retrieval systems. Our experimental results on TREC data demonstrate excellent performance in all measures---evaluation of systems, retrieval of relevant documents, and generation of metasearch lists.

References

[1]

G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Croft et~al. \citesigir98, pages 282--289.

Digital Library

Google Scholar

[2]

W. B. Croft, A. Moffat, C. J. van Rijsbergen, R. Wilkinson, and J. Zobel, editors. Proceedings of the 21th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, Aug. 1998. ACM Press, New York.

Crossref

Google Scholar

[3]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119--139, Aug. 1997.

Digital Library

Google Scholar

[4]

J. Zobel. How reliable are the results of large-scale retrieval experiments? In Croft et al. {2}, pages 307--314.

Digital Library

Google Scholar

Cited By

View all

Cormack GGrossman MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Unbiased Low-Variance Estimators for Precision and Related Information Retrieval Effectiveness MeasuresProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331355(945-948)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331355
Kurland OCulpepper JCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210186
Gao NWebber WOard D(2014)Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-MaximizationProceedings of the 36th European Conference on IR Research on Advances in Information Retrieval - Volume 841610.5555/2964060.2964062(1-12)Online publication date: 13-Apr-2014
https://dl.acm.org/doi/10.5555/2964060.2964062
Show More Cited By

Index Terms

A unified model for metasearch and the efficient evaluation of retrieval systems via the hedge algorithm
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

A unified model for metasearch, pooling, and system evaluation
CIKM '03: Proceedings of the twelfth international conference on Information and knowledge management

We present a unified model which, given the ranked lists of documents returned by multiple retrieval systems in response to a given query, simultaneously solves the problems of (1) fusing the ranked lists of documents in order to obtain a high-quality ...
Building efficient and effective metasearch engines

Frequently a user's information needs are stored in the databases of multiple search engines. It is inconvenient and inefficient for an ordinary user to invoke multiple search engines and identify useful documents from the returned results. To support ...
Rank aggregation using ant colony approach for metasearch

Metasearch engines provide a plethora of information to the user through World Wide Web. They are the prominent sources of query-based search and centralized human---world interactions. Metasearch engine shows a list of Web sites to a particular query ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval

July 2003

490 pages

ISBN:1581136463

DOI:10.1145/860435

General Chairs:
Charles Clarke
University of Waterloo, Canada
,
Gordon Cormack
University of Waterloo, Canada
,
Program Chairs:
Jamie Callan
Carnegie Mellon University, Pittsburgh, PA
,
David Hawking
Australian National University, Australia
,
Alan Smeaton
Dublin City University, Ireland

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2003

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR03

Sponsor:

SIGIR

SIGIR03: The 26th ACM/SIGIR International Symposium on Information Retrieval

July 28 - August 1, 2003

Toronto, Canada

Acceptance Rates

SIGIR '03 Paper Acceptance Rate 46 of 266 submissions, 17%;

Overall Acceptance Rate 705 of 3,463 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
521
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Cormack GGrossman MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Unbiased Low-Variance Estimators for Precision and Related Information Retrieval Effectiveness MeasuresProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331355(945-948)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331355
Kurland OCulpepper JCollins-Thompson KMei QDavison BLiu YYilmaz E(2018)Fusion in Information RetrievalThe 41st International ACM SIGIR Conference on Research & Development in Information Retrieval10.1145/3209978.3210186(1383-1386)Online publication date: 27-Jun-2018
https://dl.acm.org/doi/10.1145/3209978.3210186
Gao NWebber WOard D(2014)Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-MaximizationProceedings of the 36th European Conference on IR Research on Advances in Information Retrieval - Volume 841610.5555/2964060.2964062(1-12)Online publication date: 13-Apr-2014
https://dl.acm.org/doi/10.5555/2964060.2964062
Gao NWebber WOard D(2014)Reducing Reliance on Relevance Judgments for System Comparison by Using Expectation-MaximizationAdvances in Information Retrieval10.1007/978-3-319-06028-6_1(1-12)Online publication date: 2014
https://doi.org/10.1007/978-3-319-06028-6_1
Rajput SEkstrand-Abueg MPavlu VAslam JChen XLebanon GWang HZaki M(2012)Constructing test collections by inferring document relevance via extracted relevant informationProceedings of the 21st ACM international conference on Information and knowledge management10.1145/2396761.2396783(145-154)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2396761.2396783
Aslam JKanoulas EPavlu VSavev SYilmaz EAllan JAslam JSanderson MZhai CZobel J(2009)Document selection methodologies for efficient and effective learning-to-rankProceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval10.1145/1571941.1572022(468-475)Online publication date: 19-Jul-2009
https://dl.acm.org/doi/10.1145/1571941.1572022
Yilmaz EAslam J(2008)Estimating average precision when judgments are incompleteKnowledge and Information Systems10.5555/3227237.322750616:2(173-211)Online publication date: 1-Aug-2008
https://dl.acm.org/doi/10.5555/3227237.3227506
Pickens JGolovchinsky GShanahan JAmer-Yahia SManolescu IZhang YEvans DKolcz AChoi KChowdury A(2008)Ranked feature fusion models for ad hoc retrievalProceedings of the 17th ACM conference on Information and knowledge management10.1145/1458082.1458200(893-900)Online publication date: 26-Oct-2008
https://dl.acm.org/doi/10.1145/1458082.1458200
Pickens JGolovchinsky GShah CQvarfordt PBack MChua TLeong MMyaeng SOard DSebastiani F(2008)Algorithmic mediation for collaborative exploratory searchProceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval10.1145/1390334.1390389(315-322)Online publication date: 20-Jul-2008
https://dl.acm.org/doi/10.1145/1390334.1390389

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A unified model for metasearch, pooling, and system evaluation

Building efficient and effective metasearch engines

Rank aggregation using ant colony approach for metasearch