More Web Proxy on the site http://driver.im/

research-article

Multi-aspect expertise matching for review assignment

Authors:

Maryam Karimzadehgan,

ChengXiang Zhai,

Geneva BelfordAuthors Info & Claims

CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management

Pages 1113 - 1122

https://doi.org/10.1145/1458082.1458230

Published: 26 October 2008 Publication History

Abstract

Review assignment is a common task that many people such as conference organizers, journal editors, and grant administrators would have to do routinely. As a computational problem, it involves matching a set of candidate reviewers with a paper or proposal to be reviewed. A common deficiency of all existing work on solving this problem is that they do not consider the multiple aspects of topics or expertise and all match the entire document to be reviewed with the overall expertise of a reviewer. As a result, if a document contains multiple subtopics, which often happens, existing methods would not attempt to assign reviewers to cover all the subtopics; instead, it is quite possible that all the assigned reviewers would cover the major subtopic quite well, but not covering any other subtopic. In this paper, we study how to model multiple aspects of expertise and assign reviewers so that they together can cover all subtopics in the document well. We propose three general strategies for solving this problem and propose new evaluation measures for this task. We also create a multi-aspect review assignment test set using ACM SIGIR publications. Experiment results on this data set show that the proposed methods are effective for assigning reviewers to cover all topical aspects of a document.

References

[1]

K. Balog, L. Azzopardi, and M. de Rijke. Formal models for expert finding in enterprise corpora. In Proceedings of ACM SIGIR'06, pages 43--50, 2006.

Digital Library

[2]

C. Basu, H. Hirsh, W. Cohen, and C. Nevill-Manning. Recommending papers by mining the web. Proceedings of IJCAI Workshops on Learning About Users and Machine Learning for Information Filtering, 1999.

[3]

D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learning Res., 3:993--1022, 2003.

Digital Library

[4]

J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of SIGIR 1998, pages 335--336, 1998.

Digital Library

[5]

T. Cover and J. Thomas. Elements of Information Theory. John Wiley and Sons, New York, USA, 1991.

Digital Library

[6]

S. Dumais and J. Nielsen. Automating the assignments of submitted manuscripts to reviewers. In Proceedings of SIGIR 1992, pages 233--244, 1992.

Digital Library

[7]

H. Fang and C. Zhai. Probabilistic models for expert finding. In Proceedings of the 29th European Conference on Information Retrieval, pages 418--430, 2007.

Digital Library

[8]

M. A. Hearst. Texttiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33--64, 1997.

Digital Library

[9]

S. Hettich and M. J. Pazzani. Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD, pages 862--871, 2006.

Digital Library

[10]

T. Hofmann. Probabilistic latent sematic indexing. In Proceedings of ACM SIGIR'99, pages 50--57, 1999.

Digital Library

[11]

K. Jarvelin and J. Kekalainen. Ir evaluation methods for retrieving highly relevant documents. In Proceedings of ACM SIGIR 2000, pages 41--48, 2000.

Digital Library

[12]

B. Larson and A. Chinatsu. Fast and effective text mining using linear-time document clustering. In Proceedings of ACM SIGKDD'99, pages 16--22, 1999.

Digital Library

[13]

D. Mimno and A. McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD'07, pages 500--509, 2007.

Digital Library

[14]

D. Petkova and W. B. Croft. Hierarchical language models for expert finding in enterprise corpora. In Proceedings of the 18th IEEE on Tools with Artificial Intelligence, pages 599--608, 2006.

Digital Library

[15]

M. Rosen-Zvi, T. Griffiths, P. Smyth, and M. Steyvers. The author-topic model for authors and documents. In Proceedings of the 20th conference on Uncertainty in Artificial Intelligence, pages 487--494, 2004.

Digital Library

[16]

C. Zhai, W. Cohen, and J. Lafferty. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. Proceedings of the 26th annual international ACM SIGIR, pages 10--17, 2003.

Digital Library

[17]

C. Zhai and J. Lafferty. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of CIKM'01, pages 403--410, 2001.

Digital Library

[18]

C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of ACM Transactions on Information Systems, pages 179--214, 2004.

Digital Library

Cited By

Zheng XLi PWu X(2025)An Automatic Paper-Reviewer Recommendation Algorithm Based on Depth and BreadthIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.34026949:1(607-616)Online publication date: Feb-2025
https://doi.org/10.1109/TETCI.2024.3402694
Fu YLuo JNan GLi D(2025)Peer review expert group recommendationExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125971264:COnline publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1016/j.eswa.2024.125971
Azad SHoque MRimon NHabib MMahmud MKaiser MKarim M(2023)A Reviewer Recommender System for Scientific Articles Using a New Similarity Threshold Discovery TechniqueProceedings of the Fourth International Conference on Trends in Computational and Cognitive Engineering10.1007/978-981-19-9483-8_42(503-518)Online publication date: 28-May-2023
https://doi.org/10.1007/978-981-19-9483-8_42
Show More Cited By

Index Terms

Multi-aspect expertise matching for review assignment
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Expertise modeling for matching papers with reviewers
KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining

An essential part of an expert-finding task, such as matching reviewers to submitted papers, is the ability to model the expertise of a person based on documents. We evaluate several measures of the association between an author in an existing ...
Constrained multi-aspect expertise matching for committee review assignment
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Automatic review assignment can significantly improve the productivity of many people such as conference organizers, journal editors and grant administrators. Most previous works have set the problem up as using a paper as a query to independently "...
A robust model for paper reviewer assignment
RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

Automatic expert assignment is a common problem encountered in both industry and academia. For example, for conference program chairs and journal editors, in order to collect "good" judgments for a paper, it is necessary for them to assign the paper to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management

October 2008

1562 pages

ISBN:9781595939913

DOI:10.1145/1458082

General Chair:
James G. Shanahan
Church and Duncan Group Inc, USA
,
Program Chairs:
Sihem Amer-Yahia
Yahoo! Research, USA
,
Ioana Manolescu
INRIA, France
,
Yi Zhang
University of California, Santa Cruz, USA
,
David A. Evans
JustSystems Evans Research, USA
,
Alek Kolcz
Microsoft Live Labs, USA
,
Key-Sun Choi
KAIST, Korea
,
Abdur Chowdury
Twitter, USA

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM08

Sponsor:

CIKM08: Conference on Information and Knowledge Management

October 26 - 30, 2008

California, Napa Valley, USA

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
667
Total Downloads

Downloads (Last 12 months)35
Downloads (Last 6 weeks)2

Reflects downloads up to 22 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zheng XLi PWu X(2025)An Automatic Paper-Reviewer Recommendation Algorithm Based on Depth and BreadthIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.34026949:1(607-616)Online publication date: Feb-2025
https://doi.org/10.1109/TETCI.2024.3402694
Fu YLuo JNan GLi D(2025)Peer review expert group recommendationExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125971264:COnline publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1016/j.eswa.2024.125971
Azad SHoque MRimon NHabib MMahmud MKaiser MKarim M(2023)A Reviewer Recommender System for Scientific Articles Using a New Similarity Threshold Discovery TechniqueProceedings of the Fourth International Conference on Trends in Computational and Cognitive Engineering10.1007/978-981-19-9483-8_42(503-518)Online publication date: 28-May-2023
https://doi.org/10.1007/978-981-19-9483-8_42
Danassis PTriastcyn AFaltings BPelachaud CTaylor MFaliszewski PMascardi V(2022)A Distributed Differentially Private Algorithm for Resource Allocation in Unboundedly Large SettingsProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems10.5555/3535850.3535888(327-335)Online publication date: 9-May-2022
https://dl.acm.org/doi/10.5555/3535850.3535888
Ostendorff MBlume TRuas TGipp BRehm GAizawa AMandl TCarevic ZHinze AMayr PSchaer P(2022)Specialized document embeddings for aspect-based similarity of research papersProceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries10.1145/3529372.3530912(1-12)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3529372.3530912
Xu LZeng DDai JGui L(2022)Combining Coverage with TMPS for Reviewer Assignment2022 International Conference on Intelligent Education and Intelligent Research (IEIR)10.1109/IEIR56323.2022.10050060(107-113)Online publication date: 18-Dec-2022
https://doi.org/10.1109/IEIR56323.2022.10050060
Zhao YAnand ASharma G(2022)Reviewer Recommendations Using Document Vector Embeddings and a Publisher Database: Implementation and EvaluationIEEE Access10.1109/ACCESS.2022.315164010(21798-21811)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3151640
Liu XWang XZhu D(2022)Reviewer recommendation method for scientific research proposals: a case for NSFCScientometrics10.1007/s11192-022-04389-4127:6(3343-3366)Online publication date: 14-May-2022
https://doi.org/10.1007/s11192-022-04389-4
Kwatra DBhatnagar V(2022)Expertise Computation for Automatic Reviewer AssignmentInformation Integration and Web Intelligence10.1007/978-3-031-21047-1_48(531-547)Online publication date: 20-Nov-2022
https://doi.org/10.1007/978-3-031-21047-1_48
Stelmakh IShah NSingh A(2021)PeerReview4AllThe Journal of Machine Learning Research10.5555/3546258.354642122:1(7393-7458)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.5555/3546258.3546421
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten