More Web Proxy on the site http://driver.im/

article

Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls

Authors:

Pavel Atanasov,

Phillip Rescober,

Samuel A. Swift,

Emile Servan-Schreiber,

Philip Tetlock,

Barbara MellersAuthors Info & Claims

Management Science, Volume 63, Issue 3

Pages 691 - 706

https://doi.org/10.1287/mnsc.2015.2374

Published: 01 March 2017 Publication History

Abstract

We report the results of the first large-scale, long-term, experimental test between two crowdsourcing methods: prediction markets and prediction polls. More than 2,400 participants made forecasts on 261 events over two seasons of a geopolitical prediction tournament. Forecasters were randomly assigned to either prediction markets continuous double auction markets in which they were ranked based on earnings, or prediction polls in which they submitted probability judgments, independently or in teams, and were ranked based on Brier scores. In both seasons of the tournament, prices from the prediction market were more accurate than the simple mean of forecasts from prediction polls. However, team prediction polls outperformed prediction markets when forecasts were statistically aggregated using temporal decay, differential weighting based on past performance, and recalibration. The biggest advantage of prediction polls was at the beginning of long-duration questions. Results suggest that prediction polls with proper scoring feedback, collaboration features, and statistical aggregation are an attractive alternative to prediction markets for distilling the wisdom of crowds.

This paper was accepted by Uri Gneezy, behavioral economics.

References

[1]

Armstrong S (2001) Combining forecasts. Armstrong JS, ed. Principles of Forecasting (Springer, New York), 417-439.

[2]

Atanasov P, Rescober P, Servan-Schreiber E, Mellers B, Tetlock P, Ungar L (2013) The marketcast method for aggregating prediction market forecasts. Proc. 2013 Internat. Conf. Social Comput., Behavioral-Cultural Modeling, and Prediction (Springer, New York).

[3]

Baron J, Mellers BA, Tetlock PE, Stone E, Ungar LH (2014) Two reasons to make aggregated probability forecasts more extreme. Decision Anal. 11(2):133-145.

Digital Library

[4]

Berg J, Forsythe R, Nelson F, Rietz T (2001) Results from a dozen years of election futures markets research. Plott CR, Smith VL, eds. Handbook of Experimental Economics Results, Vol. 1 (North-Holland, Amsterdam), 742-751.

[5]

Brier GW (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1):1-3.

[6]

Camerer CF, Hogarth RM (1999) The effects of financial incentives in experiments: A review and capital-labor-production framework. J. Risk Uncertainty 19(1-3):7-42.

[7]

Chen Y, Chu C, Mullen T, Pennock D (2005) Information markets vs. opinion pools: An empirical comparison. Proc. 6th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 58-67.

[8]

Cowgill B, Zitzewitz E (2015) Corporate prediction markets: Evidence from Google, Ford, and Firm X. Rev. Econom. Stud. 82(4): 1309-1341.

[9]

Fama EF (1970) Efficient capital markets: A review of theory and empirical work. J. Finance 25(2):383-417.

[10]

Forsythe R, Nelson F, Neumann GR, Wright J (1992) Anatomy of an experimental political stock market. Amer. Econom. Rev. 82(5):1142-1161.

[11]

Goel S, Reeves D, Watts D, Pennock D (2010) Prediction without markets. Proc. 11th ACM Conf. Electronic Commerce (Association for Computing Machinery, New York), 357-366.

[12]

Graefe A, Armstrong J (2011) Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. Internat. J. Forecasting 27(1):183-195.

[13]

Gürkaynak RS, Wolfers J (2006) Macroeconomic derivatives: An initial analysis of market-based macro forecasts, uncertainty, and risk. NBER Working Paper 11929, National Bureau of Economic Research, Cambridge, MA.

[14]

Hanson R (2003) Combinatorial information market design. Inform. Systems Frontiers 5(1):107-119.

Digital Library

[15]

Hayek F (1945) The use of knowledge in society. Amer. Econom. Rev. 35(4):519-530.

[16]

Hertwig R, Ortmann A (2001) Experimental practices in economics: A methodological challenge for psychologists? Behav. Brain Sci. 24(3):383-403.

[17]

Lichtendahl KC Jr, Winkler RL (2007) Probability elicitation, scoring rules, and competition among forecasters. Management Sci. 53(11):1745-1755.

Digital Library

[18]

Laster D, Bennett P, Geoum IS (1999) Rational bias in macroeconomic forecasts. Quart. J. Econom. 114(1):293-318.

[19]

Maciejovsky B, Sutter M, Budescu D, Bernau P (2013) Teams make you smarter: How exposure to teams improves individual decisions in probability and reasoning tasks. Management Sci. 59(6):1255-1270.

[20]

Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J. Personality Soc. Psych. 107(2):276-299.

[21]

Manski C (2006) Interpreting the predictions of prediction markets. Econom. Lett. 91(3):425-429.

[22]

Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Tetlock P (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci. 25(5):1106-1115.

[23]

Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz SE, Ungar L, Bishop MM, Horowitz M, Merkle E, Tetlock P (2015a) The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. J. Experiment. Psychol.: Appl. 21(1):1-14.

[24]

Mellers B, Stone E, Murray T, Minster A, Rohrbaugh N, Bishop M, Chen E, et al. (2015b) Identifying and cultivating superforecasters as a method of improving probabilistic predictions. Perspect. Psychol. Sci. 10(3):267-281.

[25]

Moore DA, Swift SA, Minster A, Mellers B, Ungar L, Tetlock P, Yang H, Tenney R (2015) Confidence calibration in a multiyear geopolitical forecasting competition. Working paper, University of California, Berkeley, Berkeley.

[26]

Murphy AH (1973) A new vector partition of the probability score. J. Appl. Meteorology 12(4):595-600.

[27]

Ostrovsky M (2012) Information aggregation in dynamic markets with strategic traders. Econometrica 80(6):2595-2647.

[28]

Page L, Clemen RT (2012) Do prediction markets produce well-calibrated probability forecasts? Econom. J. 123(568):491-513.

[29]

Pennock DM, Lawrence S, Nielsen FA, Giles CL (2001) Extracting collective probabilistic forecasts from Web games. Proc. 7th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 174-183.

[30]

Plott C, Chen K (2002) Information aggregation mechanisms: Concept, design and implementation for a sales forecasting problem. Working paper, California Institute of Technology, Pasadena.

[31]

Ranjan R, Gneiting T (2008) Combining probability forecasts. Technical Report 543, University of Washington, Seattle.

[32]

Rieg R, Schoder R (2010) Forecasting accuracy: Comparing prediction markets and surveys--An experimental study. J. Prediction Markets 4(3):1-19.

[33]

Rosenbloom ES, Notz W (2006) Statistical tests of real-money versus play-money prediction markets. Electronic Markets 16(1):63-69.

[34]

Rothschild D (2009) Forecasting elections: Comparing prediction markets, polls, and their biases. Public Opinion Quart. 73(5): 895-916.

[35]

Rothschild D, Wolfers J (2010) Forecasting elections: Voter intentions versus expectations. Working paper, National Institute of Economic Research, Cambridge, MA.

[36]

Satopää VA, Jensen ST, Mellers BA, Tetlock PE, Ungar LH (2014a) Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. Ann. Appl. Statist. 8(2): 1256-1280.

[37]

Satopää VA, Baron J, Foster DP, Mellers BA, Tetlock PE, Ungar LH (2014b) Combining multiple probability predictions using a simple logit model. Internat. J. Forecasting 30(2):344-356.

[38]

Servan-Schreiber E (2007) About competitive forecasting. White paper, NewsFutures, Paris.

[39]

Servan-Schreiber E (2012) Prediction markets: Trading uncertainty for collective wisdom. Landemore H, Elster J, eds. Collective Wisdom: Principles and Mechanisms (Cambridge University Press, New York).

[40]

Servan-Schreiber E, Wolfers J, Pennock DM, Galebach B (2004) Prediction markets: Does money matter? Electronic Markets 14(3):243-251.

[41]

Sherden WA (1998) The Fortune Sellers: The Big Business of Buying and Selling Predictions (John Wiley & Sons, Hoboken, NJ).

[42]

Silver N (2012) The Signal and the Noise: Why So Many Predictions Fail--But Some Don't (Penguin, New York).

[43]

Snowberg E, Wolfers J (2010) Explaining the favorite-long shot bias: Is it risk-love or misperceptions? J. Political Econom. 118(4): 723-746.

[44]

Sonnemann U, Camerer C, Fox C, Langer T (2013) How psychological framing affects economic market prices in the lab and field. Proc. Natl. Acad. Sci. USA 110(29):11779-11784.

[45]

Spann M, Skiera B (2003) Internet-based virtual stock markets for business forecasting. Management Sci. 49(10):1310-1326.

Digital Library

[46]

Surowiecki J (2005) The Wisdom of Crowds (Anchor Books, New York). Swets J (1996) Signal Detection Theory and ROC Analysis in Psychology and Diagnostics: Collected Papers (Lawrence Erlbaum Associates, Mahwah, NJ).

[47]

Tetlock PE (2005) Expert Political Judgment: How Good Is It? How Can We Know? (Princeton University Press, Princeton, NJ).

[48]

Tetlock PE, Mellers BA, Rohrbaugh N, Chen E (2014) Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions Psych. Sci. 23(4):290-295.

[49]

Wolfers J (2009) Prediction markets: The collective knowledge of market participants. CFA Institute Conf. Proc. Quart. 26(2):37-44.

[50]

Wolfers J, Zitzewitz E (2006) Interpreting prediction market prices as probabilities. NBER Working Paper 12200, National Bureau of Economic Research, Cambridge, MA.

[51]

Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J. Royal Statist. Soc. 67(2):301-320.

Cited By

McCoy JPrelec D(2024)A Bayesian Hierarchical Model of Crowd Wisdom Based on Predicting Opinions of OthersManagement Science10.1287/mnsc.2023.495570:9(5931-5948)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1287/mnsc.2023.4955
Simon J(2024)Inefficient Forecasts at the SportsbookManagement Science10.1287/mnsc.2022.0045670:12(8583-8611)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1287/mnsc.2022.00456
Abdellaoui MBleichrodt HGutierrez C(2024)Unpacking Overconfident Behavior When Betting on OneselfManagement Science10.1287/mnsc.2021.0016570:10(7042-7061)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1287/mnsc.2021.00165
Show More Cited By

Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls
1. Applied computing
  1. Enterprise computing
  2. Law, social and behavioral sciences

Recommendations

Distilling wisdom of crowds in online communities: A novel prediction market constructed with comment posters
Abstract
Aggregating the wisdom of crowds from user-generated content in the online community can be valuable for decision-making. However, low-quality comments pose significant challenges for traditional wisdom extraction algorithms, such as prediction ...
Highlights
- We propose a prediction market constructed with online comment posters.
- The prediction market excels in stock price movement prediction.
- The prediction market performs well for quantitative trading.
- Prediction market surpasses ...
The Wisdom of Competitive Crowds

<P>When several individuals are asked to forecast an uncertain quantity, they often face implicit or explicit incentives to be the most accurate. Despite the desire to elicit honest forecasts, such competition induces forecasters to report strategically ...
The hasty wisdom of the mob

We explore predicting financial markets with social media sentiment.We test independent bullishness and bearishness metrics, granular to 2 minute intervals.Sentiment metrics selectively predict market behavior.Price volatility and trading volume are ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Management Science

Management Science Volume 63, Issue 3

March 2017

314 pages

ISSN:0025-1909

Issue’s Table of Contents

Publisher

INFORMS

Linthicum, MD, United States

Publication History

Published: 01 March 2017

Accepted: 14 September 2015

Received: 14 September 2014

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

McCoy JPrelec D(2024)A Bayesian Hierarchical Model of Crowd Wisdom Based on Predicting Opinions of OthersManagement Science10.1287/mnsc.2023.495570:9(5931-5948)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1287/mnsc.2023.4955
Simon J(2024)Inefficient Forecasts at the SportsbookManagement Science10.1287/mnsc.2022.0045670:12(8583-8611)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1287/mnsc.2022.00456
Abdellaoui MBleichrodt HGutierrez C(2024)Unpacking Overconfident Behavior When Betting on OneselfManagement Science10.1287/mnsc.2021.0016570:10(7042-7061)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1287/mnsc.2021.00165
Atanasov PKarger ETetlock PKleinberg RSaban DBergemann D(2024)Full Accuracy Scoring Accelerates the Discovery of Skilled ForecastersProceedings of the 25th ACM Conference on Economics and Computation10.1145/3670865.3673583(1133-1133)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3670865.3673583
Dong LZheng HLi LZhou C(2024)Distilling wisdom of crowds in online communitiesDecision Support Systems10.1016/j.dss.2024.114190180:COnline publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1016/j.dss.2024.114190
Song LZhu GYin X(2024)Evaluating the wisdom of scholar crowds from the perspective of knowledge diffusionScientometrics10.1007/s11192-024-05090-4129:9(5103-5139)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11192-024-05090-4
Corgnet BDeck CDeSantis MHampton KKimbrough E(2023)When Do Security Markets Aggregate Dispersed Information?Management Science10.1287/mnsc.2022.446369:6(3697-3729)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1287/mnsc.2022.4463
Witkowski JFreeman RVaughan JPennock DKrause A(2023)Incentive-Compatible Forecasting CompetitionsManagement Science10.1287/mnsc.2022.441069:3(1354-1374)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1287/mnsc.2022.4410
Dambanemuya HWachs JHorvat E(2023)Understanding (Ir)rational Herding OnlineProceedings of The ACM Collective Intelligence Conference10.1145/3582269.3615598(79-88)Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1145/3582269.3615598
Liu YWang JChen Y(2023)Surrogate Scoring RulesACM Transactions on Economics and Computation10.1145/356555910:3(1-36)Online publication date: 15-Feb-2023
https://dl.acm.org/doi/10.1145/3565559
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents