More Web Proxy on the site http://driver.im/

tutorial

Learning to Rank in Theory and Practice: From Gradient Boosting to Neural Networks and Unbiased Learning

Authors:

Claudio Lucchese,

Franco Maria Nardini,

Rama Kumar Pasumarthi,

Sebastian Bruch,

Michael Bendersky,

Harrie Oosterhuis,

Maarten de RijkeAuthors Info & Claims

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1419 - 1420

https://doi.org/10.1145/3331184.3334824

Published: 18 July 2019 Publication History

Abstract

This tutorial aims to weave together diverse strands of modern Learning to Rank (LtR) research, and present them in a unified full-day tutorial. First, we will introduce the fundamentals of LtR, and an overview of its various sub-fields. Then, we will discuss some recent advances in gradient boosting methods such as LambdaMART by focusing on their efficiency/effectiveness trade-offs and optimizations. Subsequently, we will then present TF-Ranking, a new open source TensorFlow package for neural LtR models, and how it can be used for modeling sparse textual features. Finally, we will conclude the tutorial by covering unbiased LtR -- a new research field aiming at learning from biased implicit user feedback. The tutorial will consist of three two-hour sessions, each focusing on one of the topics described above. It will provide a mix of theoretical and hands-on sessions, and should benefit both academics interested in learning more about the current state-of-the-art in LtR, as well as practitioners who want to use LtR techniques in their applications.

References

[1]

Mart'in Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, and others. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation. 265--283.

Digital Library

[2]

Aman Agarwal, Ivan Zaitsev, and Thorsten Joachims. 2018. Consistent position bias estimation without online interventions for learning-to-rank. arXiv preprint arXiv:1806.03555 (2018).

[3]

B. Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. 2010. Early exit optimizations for additive machine learned ranking systems. In 3rd ACM International Conference on Web Search and Data Mining. ACM, 411--420.

Digital Library

[4]

Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. 2015. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. Neural Information Processing Systems, Workshop on Machine Learning Systems (2015).

[5]

Domenico Dato, Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Nicola Tonellotto, and Rossano Venturini. 2016. Fast ranking with additive ensembles of oblivious and non-oblivious regression trees. ACM Transactions on Information Systems, Vol. 35, 2 (2016), Article 15.

Digital Library

[6]

Rolf Jagerman, Harrie Oosterhuis, and Maarten de Rijke. 2019. To model or to intervene: A comparison of counterfactual and online learning to rank from user interactions. In 42nd International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, (to appear).

Digital Library

[7]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In 22nd ACM International Conference on Multimedia. ACM, 675--678.

Digital Library

[8]

Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to-rank with biased feedback. In 10th ACM International Conference on Web Search and Data Mining. ACM, 781--789.

Digital Library

[9]

Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Salvatore Trani. 2018. X-CLEaVER: Learning ranking ensembles by growing and pruning trees. ACM Transactions on Intelligent Systems and Technology, Vol. 9, 6 (2018), Article 62.

Digital Library

[10]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016).

[11]

Harrie Oosterhuis and Maarten de Rijke. 2018. Differentiable unbiased online learning to rank. 27th ACM International Conference on Information and Knowledge Management. ACM, 1293--1302.

Digital Library

[12]

Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2019. TF-Ranking: Scalable TensorFlow library for learning-to-rank. In 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, (to appear).

Digital Library

[13]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Advances in Neural Information Processing Systems, AutoDiff Workshop: The Future of Gradient-Based Machine Learning Software and Techniques .

[14]

Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. arXiv preprint arXiv:1306.2597 (2013).

[15]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others. 2015. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, Vol. 115, 3 (2015), 211--252.

Digital Library

[16]

Lidan Wang, Jimmy J. Lin, and Donald Metzler. 2010. Learning to efficiently rank. In 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 138--145.

Digital Library

[17]

Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to rank with selection bias in personal search. In 41st International ACM SIGIR Conference on Research & Development in Information Retrieval . ACM, 115--124.

Digital Library

[18]

Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In 11th ACM International Conference on Web Search and Data Mining. ACM, 610 --618.

Digital Library

[19]

Zhixiang Xu, Olivier Chapelle, and Kilian Q Weinberger. 2012. The greedy miser: Learning under test-time budgets. In 29th International Conference on Machine Learning. 1175--1182.

Digital Library

[20]

Yisong Yue and Thorsten Joachims. 2009. Interactively optimizing information retrieval systems as a dueling bandits problem. In 26th Annual International Conference on Machine Learning. ACM, 1201--1208.

Digital Library

Cited By

Gupta SHager POosterhuis H(2024)Recent Advancements in Unbiased Learning to RankProceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation10.1145/3632754.3632942(145-148)Online publication date: 12-Feb-2024
https://doi.org/10.1145/3632754.3632942
Gupta SHager PHuang JVardasbi AOosterhuis HChen HDuh WHuang HKato MMothe JPoblete B(2023)Recent Advances in the Foundations and Applications of Unbiased Learning to RankProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3594247(3440-3443)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3594247
Omri SSinz CGuizzo GPanichella S(2022)Learning to rank for test case prioritizationProceedings of the 15th Workshop on Search-Based Software Testing10.1145/3526072.3527525(16-24)Online publication date: 9-May-2022
https://dl.acm.org/doi/10.1145/3526072.3527525
Show More Cited By

Index Terms

Learning to Rank in Theory and Practice: From Gradient Boosting to Neural Networks and Unbiased Learning
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

Unbiased Learning to Rank: Online or Offline?

How to obtain an unbiased ranking model by learning to rank with biased user feedback is an important research question for IR. Existing work on unbiased learning to rank (ULTR) can be broadly categorized into two groups—the studies on unbiased learning ...
Intent-Aware Propensity Estimation via Click Pattern Stratification
WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023

Counterfactual learning to rank via inverse propensity weighting is the most popular approach to train ranking models using biased implicit user feedback from logged search data. Standard click propensity estimation techniques rely on simple models of ...
Maximizing Marginal Fairness for Dynamic Learning to Rank
WWW '21: Proceedings of the Web Conference 2021

Rankings, especially those in search and recommendation systems, often determine how people access information and how information is exposed to people. Therefore, how to balance the relevance and fairness of information exposure is considered as one ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2019

1512 pages

ISBN:9781450361729

DOI:10.1145/3331184

General Chairs:
Benjamin Piwowarski
CNRS - Sorbonne Universite, France
,
Max Chevalier
Universite de Toulouse, CNRS, France
,
Eric Gaussier
Universite Grenoble Alpes, CNRS, France
,
Program Chairs:
Yoelle Maarek
Amazon Research, Israel
,
Jian-Yun Nie
University of Montreal, Canada
,
Falk Scholer
RMIT University, Australia

Copyright © 2019 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2019

Check for updates

Author Tags

Qualifiers

Tutorial

Funding Sources

Nederlandse Organisatie van Wetenschappelijk Onderzoek

Conference

SIGIR '19

Sponsor:

SIGIR

SIGIR '19: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 21 - 25, 2019

Paris, France

Acceptance Rates

SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
478
Total Downloads

Downloads (Last 12 months)32
Downloads (Last 6 weeks)5

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gupta SHager POosterhuis H(2024)Recent Advancements in Unbiased Learning to RankProceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation10.1145/3632754.3632942(145-148)Online publication date: 12-Feb-2024
https://doi.org/10.1145/3632754.3632942
Gupta SHager PHuang JVardasbi AOosterhuis HChen HDuh WHuang HKato MMothe JPoblete B(2023)Recent Advances in the Foundations and Applications of Unbiased Learning to RankProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3594247(3440-3443)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3594247
Omri SSinz CGuizzo GPanichella S(2022)Learning to rank for test case prioritizationProceedings of the 15th Workshop on Search-Based Software Testing10.1145/3526072.3527525(16-24)Online publication date: 9-May-2022
https://dl.acm.org/doi/10.1145/3526072.3527525
Cresci STrujillo AFagni TBellogín ABoratto LCena F(2022)Personalized Interventions for Online ModerationProceedings of the 33rd ACM Conference on Hypertext and Social Media10.1145/3511095.3536369(248-251)Online publication date: 28-Jun-2022
https://dl.acm.org/doi/10.1145/3511095.3536369
Wu XChen HZhao JHe LYin DChang YLewin-Eytan LCarmel DYom-Tov EAgichtein EGabrilovich E(2021)Unbiased Learning to Rank in Feeds RecommendationProceedings of the 14th ACM International Conference on Web Search and Data Mining10.1145/3437963.3441751(490-498)Online publication date: 8-Mar-2021
https://dl.acm.org/doi/10.1145/3437963.3441751
Oosterhuis HJagerman Rde Rijke M(2020)Unbiased Learning to Rank: Counterfactual and Online ApproachesCompanion Proceedings of the Web Conference 202010.1145/3366424.3383107(299-300)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366424.3383107

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents