More Web Proxy on the site http://driver.im/

research-article

Open access

Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks

Authors:

Sebastian Bruch,

Nadav Golbandi,

Michael Bendersky,

Marc NajorkAuthors Info & Claims

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

Pages 85 - 92

https://doi.org/10.1145/3341981.3344218

Published: 26 September 2019 Publication History

Abstract

While in a classification or a regression setting a label or a value is assigned to each individual document, in a ranking setting we determine the relevance ordering of the entire input document list. This difference leads to the notion of relative relevance between documents in ranking. The majority of the existing learning-to-rank algorithms model such relativity at the loss level using pairwise or listwise loss functions. However, they are restricted to univariate scoring functions, i.e., the relevance score of a document is computed based on the document itself, regardless of other documents in the list. To overcome this limitation, we propose a new framework for multivariate scoring functions, in which the relevance score of a document is determined jointly by multiple documents in the list. We refer to this framework as GSFs---groupwise scoring functions. We learn GSFs with a deep neural network architecture, and demonstrate that several representative learning-to-rank algorithms can be modeled as special cases in our framework. We conduct evaluation using click logs from one of the largest commercial email search engines, as well as a public benchmark dataset. In both cases, GSFs lead to significant performance improvements, especially in the presence of sparse textual features.

References

[1]

Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009. Diversifying Search Results. In Proc. of the 2nd ACM International Conference on Web Search and Data Mining. 5--14.

Digital Library

[2]

Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. 2018. Learning a deep listwise context model for ranking refinement. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 135-- 144.

Digital Library

[3]

Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier, Ed Chi, Elad Eban, Xiyang Luo, Alan Mackey, and Ofer Meshi. 2018. Seq2slate: Re-ranking and slate optimization with rnns. arXiv preprint arXiv:1810.02019 (2018).

[4]

Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A Neural Click Model for Web Search. In Proc. of the 25th International Conference on World Wide Web. 531--541.

Digital Library

[5]

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using gradient descent. In Proc. of the 22nd International Conference on Machine Learning. 89--96.

Digital Library

[6]

Christopher J.C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report Technical Report MSR-TR-2010--82. Microsoft Research.

[7]

Christopher J. C. Burges, Robert Ragno, and Quoc Viet Le. 2006. Learning to Rank with Nonsmooth Cost Functions. In Proc. of the 19th International Conference on Neural Information Processing Systems. 193--200.

Digital Library

[8]

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In Proc. of the 24th International Conference on Machine Learning. 129--136.

Digital Library

[9]

Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries. In Proc. of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 335--336.

Digital Library

[10]

O. Chapelle and Y. Chang. 2011. Yahoo! Learning to Rank Challenge Overview. In Proc. of the Learning to Rank Challenge. 1--24.

[11]

Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural Ranking Models with Weak Supervision. In Proc. of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 65--74.

[12]

Fernando Diaz. 2007. Regularizing query-based retrieval scores. Information Retrieval 10, 6 (2007), 531--562.

Digital Library

[13]

Bora Edizel, Amin Mantrach, and Xiao Bai. 2017. Deep Character-Level Click- Through Rate Prediction for Sponsored Search. In Proc. of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 305--314.

Digital Library

[14]

Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of Statistics 29, 5 (2001), 1189--1232.

[15]

Fredric C. Gey. 1994. Inferring Probability of Relevance Using the Method of Logistic Regression. In Proc. of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 222--231.

Digital Library

[16]

Jiafeng Guo, Yixing Fan, Qingyao Ai, andW. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval. In Proc. of the 25rd ACM International Conference on Information and Knowledge Management. 55--64.

Digital Library

[17]

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning Deep Structured Semantic Models for Web Search Using Clickthrough Data. In Proc. of the 22nd ACM International Conference on Information and Knowledge Management. 2333--2338.

Digital Library

[18]

Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 20, 4 (2002), 422--446.

Digital Library

[19]

Zhengbao Jiang, Ji-RongWen, Zhicheng Dou,Wayne Xin Zhao, Jian-Yun Nie, and Ming Yue. 2017. Learning to Diversify Search Results via Subtopic Attention. In Proc. of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 545--554.

Digital Library

[20]

Thorsten Joachims. 2002. Optimizing Search Engines Using Clickthrough Data. In Proc. of the 8th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 133--142.

Digital Library

[21]

Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proc. of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 217--226.

Digital Library

[22]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data As Implicit Feedback. In Proc. of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 154--161.

Digital Library

[23]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proc. of the 10th ACM International Conference on Web Search and Data Mining. 781--789.

Digital Library

[24]

Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval 3, 3 (2009), 225--331.

Digital Library

[25]

Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA.

[26]

Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to Match Using Local and Distributed Representations of Text forWeb Search. In Proc. of the 26th International Conference on World Wide Web. 1291--1299.

Digital Library

[27]

Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Jingfang Xu, and Xueqi Cheng. 2017. DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval. In Proc. of the 2017 ACM Conference on Information and Knowledge Management. 257--266.

Digital Library

[28]

Rama Kumar Pasumarthi, Xuanhui Wang, Cheng Li, Sebastian Bruch, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2018. TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank. (2018). arXiv:arXiv:1812.00073

[29]

Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. (2013). arXiv:1306.2597

[30]

Tao Qin, Tie-Yan Liu, Xu-Dong Zhang, De-Sheng Wang, and Hang Li. 2008. Global ranking using continuous conditional random fields. In Proc. of the 21st International Conference on Neural Information Processing Systems. 1281--1288.

[31]

Christian P. Robert and George Casella. 2005. Monte Carlo Statistical Methods. Springer-Verlag.

Digital Library

[32]

Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. SoftRank: Optimizing Non-smooth Rank Metrics. In Proc. of the 1st International Conference on Web Search and Data Mining. 77--86.

Digital Library

[33]

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. In Proc. of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 115--124.

Digital Library

[34]

Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position Bias Estimation for Unbiased Learning to Rank in Personal Search. In Proc. of the 11th International Conference on Web Search and Data Mining. 610 --618.

Digital Library

[35]

Fen Xia, Tie-Yan Liu, JueWang,Wensheng Zhang, and Hang Li. 2008. Listwise approach to learning to rank: theory and algorithm. In Proc. of the 25th International Conference on Machine Learning. 1192--1199.

Digital Library

[36]

Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2016. Modeling Document Novelty with Neural Tensor Network for Search Result Diversification. In Proc. of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 395--404.

Digital Library

[37]

Jun Xu and Hang Li. 2007. AdaRank: A Boosting Algorithm for Information Retrieval. In Proc. of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 391--398.

Digital Library

[38]

Peng Ye and David Doermann. 2013. Combining preference and absolute judgements in a crowd-sourced setting. In ICML 2013 Workshop on Machine Learning Meets Crowdsourcing.

[39]

Hamed Zamani, Michael Bendersky, Xuanhui Wang, and Mingyang Zhang. 2017. Situational Context for Ranking in Personal Search. In Proc. of the 26th International Conference on World Wide Web. 1531--1540.

Digital Library

Cited By

Wang YLi ZZhang CChen SZhang XXu JLin Q(2024)Do Not Wait: Learning Re-Ranking Model Without User Feedback At Serving Time in E-CommerceProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688165(896-901)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688165
Zhang JYu QChen YZhou GLiu YSun YLiang CHuzhang GNi YZeng AYu HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)An E-Commerce Dataset Revealing Variations during SalesProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657870(1162-1171)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657870
Xu SPang LXu JShen HCheng XChua TNgo CKa-Wei Lee RKumar RLauw H(2024)List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented GenerationProceedings of the ACM Web Conference 202410.1145/3589334.3645336(1330-1340)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645336
Show More Cited By

Index Terms

Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

Analysis of Multivariate Scoring Functions for Automatic Unbiased Learning to Rank
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Leveraging biased click data for optimizing learning to rank systems has been a popular approach in information retrieval. Because click data is often noisy and biased, a variety of methods have been proposed to construct unbiased learning to rank (ULTR)...
A regression framework for learning ranking functions using relative relevance judgments
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Effective ranking functions are an essential part of commercial search engines. We focus on developing a regression framework for learning ranking functions for improving relevance of search engines serving diverse streams of user queries. We explore ...
Surrogate scoring for improved metasearch precision
SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval

We describe a method for improving the precision of metasearch results based upon scoring the visual features of documents' surrogate representations. These surrogate scores are used during fusion in place of the original scores or ranks provided by the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

September 2019

273 pages

ISBN:9781450368810

DOI:10.1145/3341981

General Chairs:
Yi Fang
Santa Clara University, USA
,
Yi Zhang
University of California, Santa Cruz, USA
,
Program Chairs:
James Allan
University of Massachusetts, Amherst, USA
,
Krisztian Balog
University of Stavanger, Norway
,
Ben Carterette
Spotify, USA
,
Jiafeng Guo
Chinese Academy of Sciences, China

Copyright © 2019 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 September 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICTIR '19

Sponsor:

SIGIR

ICTIR '19: The 2019 ACM SIGIR International Conference on the Theory of Information Retrieval

October 2 - 5, 2019

CA, Santa Clara, USA

Acceptance Rates

ICTIR '19 Paper Acceptance Rate 20 of 41 submissions, 49%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

47
Total Citations
View Citations
1,210
Total Downloads

Downloads (Last 12 months)220
Downloads (Last 6 weeks)19

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang YLi ZZhang CChen SZhang XXu JLin Q(2024)Do Not Wait: Learning Re-Ranking Model Without User Feedback At Serving Time in E-CommerceProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688165(896-901)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688165
Zhang JYu QChen YZhou GLiu YSun YLiang CHuzhang GNi YZeng AYu HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)An E-Commerce Dataset Revealing Variations during SalesProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657870(1162-1171)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657870
Xu SPang LXu JShen HCheng XChua TNgo CKa-Wei Lee RKumar RLauw H(2024)List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented GenerationProceedings of the ACM Web Conference 202410.1145/3589334.3645336(1330-1340)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645336
Sun HHuang XMa W(2024)Beyond Prediction: On-Street Parking Recommendation Using Heterogeneous Graph-Based List-Wise RankingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.333680825:6(5892-5903)Online publication date: Jun-2024
https://doi.org/10.1109/TITS.2023.3336808
Hassan BClough CSiddiqi YFaizan Ali RArshed M(2024)PlayerRank: Leveraging Learning-to-Rank AI for Player Positioning in CricketIEEE Access10.1109/ACCESS.2024.349552812(177504-177519)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3495528
Li YXiong HKong LBian JWang SChen GYin D(2024)GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale searchMachine Learning10.1007/s10994-023-06469-9113:8(5331-5349)Online publication date: 5-Jan-2024
https://doi.org/10.1007/s10994-023-06469-9
Nardini FTrani RVenturini R(2024)Learning bivariate scoring functions for rankingDiscover Computing10.1007/s10791-024-09444-727:1Online publication date: 27-Sep-2024
https://doi.org/10.1007/s10791-024-09444-7
Liu WZhou YZhu YDou Z(2024)How to personalize and whether to personalize? Candidate documents decideKnowledge and Information Systems10.1007/s10115-024-02138-y66:9(5581-5604)Online publication date: 27-May-2024
https://doi.org/10.1007/s10115-024-02138-y
He LLuo KDing ZShao HBai B(2024)A Reinforcement Learning Approach for Personalized Diversity in Feeds RecommendationArtificial Intelligence10.1007/978-981-99-9119-8_42(463-475)Online publication date: 3-Feb-2024
https://doi.org/10.1007/978-981-99-9119-8_42
Guo TZhang TWu HLi HQiao RSun X(2024)Multimodal Label Relevance Ranking via Reinforcement LearningComputer Vision – ECCV 202410.1007/978-3-031-72848-8_23(391-408)Online publication date: 29-Nov-2024
https://doi.org/10.1007/978-3-031-72848-8_23
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents