More Web Proxy on the site http://driver.im/

research-article

Public Access

Eliciting Expertise without Verification

Authors:

Grant SchoenebeckAuthors Info & Claims

EC '18: Proceedings of the 2018 ACM Conference on Economics and Computation

Pages 195 - 212

https://doi.org/10.1145/3219166.3219172

Published: 11 June 2018 Publication History

Abstract

A central question of crowdsourcing is how to elicit expertise from agents. This is even more difficult when answers cannot be directly verified. A key challenge is that sophisticated agents may strategically withhold effort or information when they believe their payoff will be based upon comparison with other agents whose reports will likely omit this information due to lack of effort or expertise. Our work defines a natural model for this setting based on the assumption that more sophisticated agents know the beliefs of less sophisticated agents. We then provide a mechanism design framework for this setting. From this framework, we design several novel mechanisms, for both the single and multiple tasks settings, that (1) encourage agents to invest effort and provide their information honestly; (2) output a correct "hierarchy" of the information when agents are rational.

Supplementary Material

MP4 File (p195.mp4)

Download
445.39 MB

References

[1]

Arpit Agarwal, Debmalya Mandal, David C. Parkes, and Nisarg Shah. 2017. Peer Prediction with Heterogeneous Users. In Proceedings of the 2017 ACM Conference on Economics and Computation, EC '17, Cambridge, MA, USA, June 26--30, 2017. 81--98.

Digital Library

[2]

Syed Mumtaz Ali and Samuel D Silvey. 1966. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B (Methodological) (1966), 131--142.

[3]

Y. Chen, D.F. Bacon, I. Kash, D.C. Parkes, M. Rao, and M. Sridharan. 2012. Predicting Your Own Effort. In Proceedings of the 11th International Conference on Autonomous and Multiagent Systems (AAMAS 2012).

Digital Library

[4]

Thomas M Cover and Joy A Thomas. 2006. Elements of information theory 2nd edition. (2006).

[5]

Imre Csiszár, Paul C Shields, et al. 2004. Information theory and statistics: A tutorial. Foundations and Trends® in Communications and Information Theory Vol. 1, 4 (2004), 417--528.

Digital Library

[6]

Anirban Dasgupta and Arpita Ghosh. 2013. Crowdsourced judgement elicitation with endogenous proficiency Proceedings of the 22nd international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 319--330.

Digital Library

[7]

A. P. Dawid and A. M. Skene. 1979. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) Vol. 28, 1 (1979), 20--28. Boi Faltings. 2006. Minimum payments that reward honest reputation feedback Proceedings of the 7th ACM conference on Electronic commerce (EC 2006).

Digital Library

[8]

Radu Jurca and Boi Faltings. 2007. Robust Incentive-Compatible Feedback Payments. In Trust, Reputation and Security: Theories and Practice, Vol. Vol. 4452. Springer-Verlag, 204--218.

Digital Library

[9]

Radu Jurca and Boi Faltings. 2008. Incentives for expressing opinions in online polls Proceedings of the 9th ACM conference on Electronic commerce (EC 2008).

Digital Library

[10]

Radu Jurca and Boi Faltings. 2009. Mechanisms for making crowds truthful. J. Artif. Int. Res. Vol. 34, 1 (March. 2009).

Digital Library

[11]

R. Jurca and B. Faltings. 2011. Incentives for Answering Hypothetical Questions. In Proceedings of the 1st Workshop on Social Computing and User Generated Content (SC 2011). ACM.

[12]

Vijay Kamble, Nihar Shah, David Marn, Abhay Parekh, and Kannan Ramachandran. 2015. Truth Serums for Massively Crowdsourced Evaluation Tasks. arXiv preprint arXiv:1507.07045 (2015).

[13]

Yuqing Kong, Katrina Ligett, and Grant Schoenebeck. 2016. Putting peer prediction under the micro (economic) scope and making truth-telling focal. In International Conference on Web and Internet Economics. Springer, 251--264.

Digital Library

[14]

Y. Kong and G. Schoenebeck. 2016. A Framework For Designing Information Elicitation Mechanisms That Reward Truth-telling. ArXiv e-prints (May. 2016). showeprint{arxiv}cs.GT/1605.01021

[15]

Yuqing Kong and Grant Schoenebeck. 2018. Equilibrium selection in information elicitation without verification via information monotonicity. In 9th Innovations in Theoretical Computer Science, ITCS 2018, Vol. Vol. 94.

[16]

N. Lambert and Y. Shoham. 2008. Truthful surveys. Proceedings of the 3rd International Workshop on Internet and Network Economics (WINE 2008) (2008).

Digital Library

[17]

Yang Liu and Yiling Chen. 2016. Learning to incentivize: eliciting effort via output agreement. arXiv preprint arXiv:1604.04928 (2016).

[18]

Yang Liu and Yiling Chen. 2016. Sequential Peer Prediction: Learning to Elicit Effort using Posted Prices. arXiv preprint arXiv:1611.09219 (2016).

[19]

Debmalya Mandal, Matthew Leifer, David C Parkes, Galen Pickard, and Victor Shnayder. 2016. Peer Prediction with Heterogeneous Tasks. arXiv preprint arXiv:1612.00928 (2016).

[20]

John McCoy and Drazen Prelec. 2017. A statistical model for aggregating judgments by incorporating peer predictions. arXiv preprint arXiv:1703.04778 (2017).

[21]

N. Miller, P. Resnick, and R. Zeckhauser. 2005. Eliciting informative feedback: The peer-prediction method. Management Science (2005), 1359--1373.

Digital Library

[22]

D. Prelec. 2004. A Bayesian Truth Serum for subjective data. Science Vol. 306, 5695 (2004), 462--466.

[23]

Dravzen Prelec, H Sebastian Seung, and John McCoy. 2017. A solution to the single-question crowd wisdom problem. Nature Vol. 541, 7638 (2017), 532--535.

[24]

Goran Radanovic and Boi Faltings. 2013. A robust bayesian truth serum for non-binary signals Proceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI 2013. 833--839.

Digital Library

[25]

Goran Radanovic and Boi Faltings. 2014. Incentives for truthful information elicitation of continuous signals Twenty-Eighth AAAI Conference on Artificial Intelligence.

Digital Library

[26]

Goran Radanovic and Boi Faltings. 2015. Incentive schemes for participatory sensing. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. 1081--1089.

Digital Library

[27]

Goran Radanovic and Boi Faltings. 2015. Incentives for Subjective Evaluations with Private Beliefs Twenty-Ninth AAAI Conference on Artificial Intelligence.

Digital Library

[28]

Blake Riley. 2014. Minimum truth serums with optional predictions. In Proceedings of the 4th Workshop on Social Computing and User Generated Content (SC14).

[29]

Blake Riley. 2015. Mechanisms for Making Accurate Decisions in Biased Crowds. (2015).

[30]

Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C Parkes. 2016. Informed truthfulness in multi-task peer prediction Proceedings of the 2016 ACM Conference on Economics and Computation. ACM, 179--196.

Digital Library

[31]

Peter Welinder, Steve Branson, Serge J Belongie, and Pietro Perona. 2010. The Multidimensional Wisdom of Crowds. In NIPS, Vol. Vol. 23. 2424--2432.

Digital Library

[32]

Robert L Winkler. 1969. Scoring rules and the evaluation of probability assessors. J. Amer. Statist. Assoc. Vol. 64, 327 (1969), 1073--1078.

[33]

Jens Witkowski, Bernhard Nebel, and David C Parkes. 2014. Robust Peer Prediction Mechanisms. Ph.D. Dissertation. Ph. D. Dissertation, Albert-Ludwigs-Universitat Freiburg: Institut fur Informatik.

[34]

Jens Witkowski and David C Parkes. 2012. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce. ACM, 964--981.

Digital Library

[35]

Jens Witkowski and David C Parkes. 2013. Learning the prior in minimal peer prediction. In Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce. Citeseer, 14.

[36]

Peter Zhang and Yiling Chen. 2014. Elicitability and knowledge-free elicitation with peer prediction Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 245--252.

Digital Library

[37]

Denny Zhou, Sumit Basu, Yi Mao, and John C Platt. 2012. Learning from the wisdom of crowds by minimax entropy Advances in Neural Information Processing Systems. 2195--2203.

Digital Library

Cited By

Lu YXu SZhang YKong YSchoenebeck GKleinberg RSaban DBergemann D(2024)Eliciting Informative Text Evaluations with Large Language ModelsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673532(582-612)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3670865.3673532
Kong Y(2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1145/3638239
Xu SZhang YResnick PSchoenebeck GChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Spot Check Equivalence: An Interpretable Metric for Information Elicitation MechanismsProceedings of the ACM Web Conference 202410.1145/3589334.3645679(276-287)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645679
Show More Cited By

Index Terms

Eliciting Expertise without Verification
1. Information systems
  1. World Wide Web
    1. Web applications
      1. Crowdsourcing
        Incentive schemes
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Algorithmic game theory and mechanism design
      1. Algorithmic mechanism design

Recommendations

Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of Tasks
¹ In the setting where participants are asked multiple similar possibly subjective multi-choice questions (e.g., Do you like Panda Express? Y/N; Do you like Chick-fil-A? Y/N), a series of peer prediction mechanisms have been designed to incentivize honest ...
Two Strongly Truthful Mechanisms for Three Heterogeneous Agents Answering One Question
Peer prediction mechanisms incentivize self-interested agents to truthfully report their signals even in the absence of verification by comparing agents’ reports with their peers. We propose two new mechanisms, Source and Target Differential Peer ...
Water from Two Rocks: Maximizing the Mutual Information
EC '18: Proceedings of the 2018 ACM Conference on Economics and Computation

We build a natural connection between the learning problem, co-training, and forecast elicitation without verification (related to peer-prediction) and address them simultaneously using the same information theoretic approach. In co-training/multiview ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

EC '18: Proceedings of the 2018 ACM Conference on Economics and Computation

June 2018

713 pages

ISBN:9781450358293

DOI:10.1145/3219166

General Chair:
Eva Tardos
Cornell University, USA
,
Program Chairs:
Edith Elkind
University of Oxford, UK
,
Rakesh Vohra
University of Pennsylvania, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGecom: Special Interest Group on Economics and Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

EC '18

Sponsor:

SIGecom

EC '18: ACM Conference on Economics and Computation

June 18 - 22, 2018

NY, Ithaca, USA

Acceptance Rates

EC '18 Paper Acceptance Rate 70 of 269 submissions, 26%;

Overall Acceptance Rate 664 of 2,389 submissions, 28%

Upcoming Conference

EC '25

Sponsor:
sigecom

The 25th ACM Conference on Economics and Computation

July 7 - 11, 2025

Stanford , CA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
409
Total Downloads

Downloads (Last 12 months)114
Downloads (Last 6 weeks)19

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lu YXu SZhang YKong YSchoenebeck GKleinberg RSaban DBergemann D(2024)Eliciting Informative Text Evaluations with Large Language ModelsProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673532(582-612)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3670865.3673532
Kong Y(2024)Dominantly Truthful Peer Prediction Mechanisms with a Finite Number of TasksJournal of the ACM10.1145/363823971:2(1-49)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1145/3638239
Xu SZhang YResnick PSchoenebeck GChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Spot Check Equivalence: An Interpretable Metric for Information Elicitation MechanismsProceedings of the ACM Web Conference 202410.1145/3589334.3645679(276-287)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645679
Zhang YSchoenebeck G(2023)High-Effort Crowds: Limited Liability via TournamentsProceedings of the ACM Web Conference 202310.1145/3543507.3583334(3467-3477)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583334
Kong YLi YZhang YHuang ZWu JKoyejo SMohamed SAgarwal ABelgrave DCho KOh A(2022)Eliciting thinking hierarchy without a priorProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3601239(13329-13341)Online publication date: 28-Nov-2022
https://dl.acm.org/doi/10.5555/3600270.3601239
Kong YChawla S(2020)Dominantly truthful multi-task peer prediction with a constant number of tasksProceedings of the Thirty-First Annual ACM-SIAM Symposium on Discrete Algorithms10.5555/3381089.3381236(2398-2411)Online publication date: 5-Jan-2020
https://dl.acm.org/doi/10.5555/3381089.3381236
Schoenebeck GYu F(2020)Two Strongly Truthful Mechanisms for Three Heterogeneous Agents Answering One QuestionWeb and Internet Economics10.1007/978-3-030-64946-3_9(119-132)Online publication date: 6-Dec-2020
https://doi.org/10.1007/978-3-030-64946-3_9
Kong YSchoenebeck G(2019)An Information Theoretic Framework For Designing Information Elicitation Mechanisms That Reward Truth-tellingACM Transactions on Economics and Computation10.1145/32966707:1(1-33)Online publication date: 25-Jan-2019
https://dl.acm.org/doi/10.1145/3296670
Kong YSchoenebeck GTardos EElkind EVohra R(2018)Water from Two RocksProceedings of the 2018 ACM Conference on Economics and Computation10.1145/3219166.3219194(177-194)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3219166.3219194

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents