Abstract
Expert elicitation plays a prominent role in fields where the data are scarce. As consulting multiple experts is critical in expert elicitation practices, combining various expert opinions is an important topic. In the Classical Model, uncertainty distributions for the variables of interest are based on an aggregation of elicited expert percentiles. Aggregation of these expert distributions is accomplished using linear opinion pooling relying on performance-based weights that are assigned to each expert. According to the Classical Model, each expert receives a weight that is a combination of the expert’s statistical accuracy and informativeness for a set of questions, the values of which are unknown at the time the elicitation was conducted. The former measures “correspondence with reality,” a measure of discrepancy between the observed relative frequencies of seed variables’ values falling within the elicited percentile values and the expected probability based on the percentiles specified in the elicitation. The later gauges an expert’s ability to concentrate high probability mass in small interquartile intervals. Some critics argue that this performance-based model fails to outperform the models that assign experts equal weights. Their argument implies that any observed difference in expert performance is just due to random fluctuations and is not a persistent property of an expert. Experts should therefore be treated equally and equally weighted. However, if differences in experts’ performances are due to random fluctuations, then hypothetical experts created by randomly recombining the experts’ assessments should perform statistically as well as the actual experts. This hypothesis is called the random expert hypothesis. This hypothesis is investigated using 44 post-2006 professional expert elicitation studies obtained through the TU Delft database. For each study, 1000 hypothetical expert panels are simulated whose elicitations are a random mix of all expert elicitations within that study. Results indicate that actual expert statistical accuracy performance is significantly better than that of randomly created experts. The study does not consider experts’ informativeness but still provides strong support for performance-based weighting as in the Classical Model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aspinall, W. (2010). A route to more tractable expert advice. Nature, 463(7279), 294.
Aspinall, W. P., Cooke, R. M., Havelaar, A. H., Hoffmann, S., & Hald, T. (2016). Evaluation of a performance-based expert elicitation: WHO global attribution of foodborne diseases. PLoS ONE, 11(3), e0149817.
Bamber, J. L., & Aspinall, W. P. (2013). An expert judgement assessment of future sea level rise from the ice sheets. Nature Climate Change, 3(4), 424.
Bamber, J. L., Aspinall, W. P., & Cooke, R. M. (2016). A commentary on “how to interpret expert judgment assessments of twenty-first century sea-level rise” by Hylke de Vries and Roderik SW van de Wal. Climatic Change, 137(3–4), 321–328.
Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5(4), 559–583.
Clemen, R. T. (2008). Comment on Cooke’s classical method. Reliability Engineering & System Safety, 93(5), 760–765.
Clemen, R. T., & Winkler, R. L. (1999). Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2), 187–203.
Colson, A. R., & Cooke, R. M. (2017). Cross validation for the classical model of structured expert judgment. Reliability Engineering & System Safety, 163, 109–120.
Cooke, R. (1991). Experts in uncertainty: Opinion and subjective probability in science. Oxford University Press on Demand.
Cooke, R. M., & Goossens, L. L. (2008). TU Delft expert judgment data base. Reliability Engineering & System Safety, 93(5), 657–674.
Eggstaff, J. W., Mazzuchi, T. A., & Sarkani, S. (2014). The effect of the number of seed variables on the performance of Cooke′ s classical model. Reliability Engineering & System Safety, 121, 72–82.
Einhorn, H. J. (1974). Expert judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59(5), 562.
French, S. (1981). Consensus of opinion. European Journal of Operational Research, 7(4), 332–340.
Genest, C., & Zidek, J. V. (1986). Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1(1), 114–135.
Hald, T., Aspinall, W., Devleesschauwer, B., Cooke, R., Corrigan, T., Havelaar, A. H., et al. (2016). World Health Organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: A structured expert elicitation. PLoS ONE, 11(1), e0145839.
Jaiswal, K. S., Aspinall, W., Perkins, D., Wald, D., & Porter, K. A. (2012). Use of expert judgment elicitation to estimate seismic vulnerability of selected building types. In 15th World Conference on Earthquake Engineering (WCEE). Lisbon, Portugal, Sept (pp. 24–28).
Keeney, R. L., & Von Winterfeldt, D. (1989). On the uses of expert judgment on complex technical problems. IEEE Transactions on Engineering Management, 36(2), 83–86.
Morgan, M. G., Henrion, M., & Small, M. (1992). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge university press.
Mosleh, A., Bier, V. M., & Apostolakis, G. (1988). A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering & System Safety, 20(1), 63–85.
Otway, H., & von Winterfeldt, D. (1992). Expert judgment in risk analysis and management: process, context, and pitfalls. Risk Analysis, 12(1), 83–93.
Ouchi, F. (2004). A literature review on the use of expert opinion in probabilistic risk analysis.
Ryan, J. J., Mazzuchi, T. A., Ryan, D. J., De la Cruz, J. L., & Cooke, R. (2012). Quantifying information security risks using expert judgment elicitation. Computers & Operations Research, 39(4), 774–784.
Singpurwalla, N. D. (1988). Foundational issues in reliability and risk analysis. SIAM Review, 30(2), 264–282.
Spetzler, C. S., & Stael von Holstein, C. A. S. (1975). Exceptional paper—probability encoding in decision analysis. Management Science, 22(3), 340–358.
Stone, M. (1961). The opinion pool. The Annals of Mathematical Statistics, 1339–1342.
Tyshenko, M. G., ElSaadany, S., Oraby, T., Darshan, S., Aspinall, W., Cooke, R., et al. (2011). Expert elicitation for the judgment of prion disease risk uncertainties. Journal of Toxicology and Environmental Health, Part A, 74(2–4), 261–285.
Wallsten, T. S., & Budescu, D. V. (1983). State of the art—encoding subjective probabilities: A psychological and psychometric review. Management Science, 29(2), 151–173.
Wilson, K. J. (2017). An investigation of dependence in expert judgement studies with multiple experts. International Journal of Forecasting, 33(1), 325–336.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
Data references table
Study name | References |
|---|---|
UMD | Koch, Benjamin J., Filoso, S., Cooke, R. M. Hosen, J. D., Colson, A.R. Febria, Catherine M., Palmer, M. A., (2015) Nitrogen in stormwater runoff from Coastal Plain watersheds: The need for empirical data, reply to Walsh, Elementa DOI https://doi.org/10.12952/journal.elementa.000079. https://www.elementascience.org/articles/79 Koch, Benjamin J., Febria, Catherine M., Cooke, Roger M. Hosen, Jacob D., Baker, Matthew E., Colson, Abigail R. Filoso, Solange, Hayhoe, Katharine, Loperfido, J.V., Stoner, Anne M.K., Palmer, Margaret A., (2015) Suburban watershed nitrogen retention: Estimating the effectiveness of storm water management structures, Elementa, https://doi.org/10.12952/journal.elementa.000063 https://www.elementascience.org/articles/63 |
USGS | Newhall, C. G., & Pallister, J. S. (2015). Using multiple data sets to populate probabilistic volcanic event trees. In Volcanic Hazards, Risks and Disasters (pp. 203–232) |
arsenic | Hanzich, J.M. (2007) Achieving Consensus: An Analysis Of Methods To Synthesize Epidemiological Data For Use In Law And Policy. Department of Public Health & Primary Care, Institute Of Public Health, University of Cambridge; unpublished MPhil thesis, 66 pp + appendices |
Biol Agents | Aspinall & Associates (2006). REBA Elicitation. Commercial-in-confidence report, pp. 26 |
Geopolit | Ismail and Reid (2006). “Ask the Experts” presentation |
ATCEP | Morales-Nápoles, O., Kurowicka, D., & Cooke, R. (2008). EEMCS final report for the causal modeling for air transport safety (CATS) project |
Daniela | Forys, M.B., Kurowicka, D., Peppelman, B.(2013) “A probabilistic model for a gas explosion due to leakages in the grey cast iron gas mains” Reliability Engineering & System Safety volume 119, issue, year 2013, pp. 270–279 |
eBBP | Tyshenko, M.G., S. ElSaadany, T. Oraby, M. Laderoute, J. Wu, W. Aspinall and D. Krewski (2011) Risk Assessment and Management of Emerging Blood-Borne Pathogens in Canada: Xenotropic Murine Leukaemia Virus-Related Virus as a Case Study for the Use of a Precautionary Approach. Chapter in: Risk Assessment (ISBN 979-953-307-765-8) Cashman, N.R., Cheung, R., Aspinall, W., Wong, M. and Krewski, D. (2014) Expert Elicitation for the Judgment of Prion Disease Risk Uncertainties associated with Urine-derived and Recombinant Fertility Drugs. Submitted to: Journal of Toxicology and Environmental Health |
create | Bier V.M, Kosanoglu, F, Shin J, unpublished data, nd |
effErupt | Aspinall, W.P. (2012) Comment on “Social studies of volcanology: knowledge generation and expert advice on active volcanoes” by Amy Donovan, Clive Oppenheimer and Michael Bravo [Bull Volcanol (2012) 74:677–689] Bulletin of Volcanology, 74, 1569–1570. https://doi.org/10.1007/s00445-012-0625-x |
erie | Colson, Abigail R., Sweta Adhikari, Ambereen Sleemi, and Ramanan Laxminarayan. (2015) “Quantifying Uncertainty in Intervention Effectiveness with Structured Expert Judgment: An Application to Obstetric Fistula.” BMJ Open, 1–8. https://doi.org/10.1136/bmjopen-2014-007233 Cooke, R.M., Wittmann, M.E., Lodge, D.M., Rothlisberger, J.D., Rutherford E.S., Zhang, H. and Mason, D.M. (2014) “Out-of-Sample Validation for Structured Expert Judgment of Asian Carp Establishment in Lake Erie”, Integrated Environmental Assessment and Management, open access. DOI: https://doi.org/10.1002/ieam.1559 Zhang, H, Rutherford E.S., Mason, D.M., Breck, J,T,, Wittmann M.E., Cooke R.M., Lodge D.M., Rothlisberger J.D., Zhu X., and Johnson, T B., (2015) Forecasting the Impacts of Silver and Bighead Carp on the Lake Erie Food Web, Transactions of the American Fisheries Society, Volume 145, Issue 1, pp 136–162, https://doi.org/10.1080/00028487.2015.1069211 |
FCEP | Leontaris, G., & Morales-Nápoles, O. (2018). ANDURIL—A MATLAB toolbox for ANalysis and Decisions with UnceRtaInty: Learning from expert judgments. SoftwareX, 7, 313–317 |
Sheep | Hincks, T., Aspinall, W. and Stone, J. (2015) Expert judgement elicitation exercise to evaluate Sheep Scab control measures: Results of the Bayesian Belief Network analysis. University of Bristol PURE Repository Working Paper (forthcoming) |
hemophilia | Fischer K, Lewandowski D, Janssen MP. Estimating unknown parameters in haemophilia using expert judgement elicitation. Haemophilia. 2013 Sep;19(5):e282–e288 |
Liander | Forys, M.B., Kurowicka, D., Peppelman, B.(2013) “A probabilistic model for a gas explosion due to leakages in the grey cast iron gas mains” Reliability Engineering & System Safety volume 119, issue, year 2013, pp. 270–279 |
PHAC | Oraby,T., Tyshenko, M.G., Westphal, M., Darshan, S., Croteau, M., Aspinall, W., Elsaadany, S., Cashman, N. and Krewski, D. (2011) Using Expert Judgments to Improve Chronic Wasting Disease Risk Management in Canada. Journal of Toxicology and Environmental Health, in press. Volume 74, Issue 2-4, 2011 Special Issue: Prion Research in Perspective 2010 |
TOPAZ | Scourse, E., Aspinall, W.P. and Chapman, N. (2014) Using expert elicitation to characterise long-term tectonic risks to radioactive waste repositories in Japan. Journal of Risk Research, https://doi.org/10.1080/13669877.2014.971334 |
SPEED | Hicks, A., Barclay, J., Simmons, P. and Loughlin, S. (2014). “An interdisciplinary approach to volcanic risk reduction under conditions of uncertainty: a case study of Tristan da Cunha.” Nat. Hazards Earth Syst. Sci. 14(7): 1871-1887. https://doi.org/10.5194/nhess-14-1871-2014. www.nat-hazards-earth-syst-sci-discuss.net/1/7779/2013/ Bevilacqua, A., Isaia, R., Neri, A., Vitale, S., Aspinall, W.P. and eight others (2015) Quantifying volcanic hazard at Campi Flegrei caldera (Italy) with uncertainty assessment: I. Vent opening maps. Journal of Geophysical Research—Solid Earth; AGU. https://doi.org/10.1002/2014jb011775 |
TDC | Scourse, E., Aspinall, W.P. and Chapman, N. (2014) Using expert elicitation to characterise long-term tectonic risks to radioactive waste repositories in Japan. Journal of Risk Research, https://doi.org/10.1080/13669877.2014.971334 |
GL | Rothlisberger,J.D. Finnoff, D.C. Cooke,R.M. and Lodge, D.M. (2012) “Ship-borne nonindigenous species diminish Great Lakes ecosystem services” Ecosystems (2012) 15: 462–476 https://doi.org/10.1007/s10021-012-9522-6 Rothlisberger, J.D., Lodge, D.M. Cooke, R.M. and Finnoff, D.C. (2009) “Future declines of the binational Laurentian Great Lakes fisheries: recognizing the importance of environmental and cultural change” Frontiers in Ecology and the Environment; https://doi.org/10.1890/090002 |
Goodheart | Goodheart, B. (2013). Identification of causal paths and prediction of runway incursion risk by means of Bayesian belief networks. Transportation Research Record: Journal of the Transportation Research Board, (2400), 9–20. |
Ice | Bamber, J.L., and Aspinall, W.P., (2012) An expert judgement assessment of future sea 1evel rise from the ice sheets, Nature Climate Change, PUBLISHED ONLINE: January 6, 2012| https://doi.org/10.1038/nclimate1778. http://www.nature.com/nclimate/journal/vaop/ncurrent/full/nclimate1778.html |
puig-gdp | Puig, D., Morales-Nápoles, O., Bakhtiari, F., & Landa, G. (2018). The accountability imperative for quantifying the uncertainty of emission forecasts: evidence from Mexico. Climate Policy, 18(6), 742–751 |
puig-oil | Puig, D., Morales-Nápoles, O., Bakhtiari, F., & Landa, G. (2018). The accountability imperative for quantifying the uncertainty of emission forecasts: evidence from Mexico. Climate Policy, 18(6), 742–751 |
YTBID (CDC) | Colson, Abigail R., M.A. Cohen, S. Regmi, A. Nandi, R. Laxminarayan (2015) “Structured Expert Judgment for Informing the Return on Investment in Surveillance: The Case of Environmental Public Health Tracking.” Working Paper. Center for Disease Dynamics, Economics & Policy |
Gerestenberger | Gerstenberger, M. C., et al. (2016). “A Hybrid Time‐Dependent Probabilistic Seismic‐Hazard Model for Canterbury, New Zealand.” Seismological Research Letters. Vol. 87 Doi: https://doi.org/10.1785/0220160084 Gerstenberger, M.C.; McVerry, G.H.; Rhoades, D.A.; Stirling, M.W. (2014) Seismic hazard modeling for the recovery of Christchurch, New Zealand.Earthquake Spectra, 30(1): 17–29; https://doi.org/10.1193/021913eqs037m Gerstenberger, M.C.; Christophersen, A.; Buxton, R.; Allinson, G.; Hou, W.; Leamon, G.; Nicol, A. (2013) Integrated risk assessment for CCS. p. 2775–2782; https://doi.org/10.1016/j.egypro.2013.06.162 IN: Dixon, T.; Yamaji, K. (eds) 11th International Conference on Greenhouse Gas Control Technologies, 18th-22nd November 2012, Kyoto International Conference Center, Japan. Elsevier. Energy procedia 37 |
CWD | Tyshenko, M.G., ElSaadany, S., Oraby, T., Darshan, S., Catford, A., Aspinall, W., Cooke, R. and Krewski, D. (2012) Expert judgement and re-elicitation for prion disease risk uncertainties. International Journal of Risk Assessment and Management, 16(1–3), 48–77. https://doi.org/10.1504/ijram.2012.047552 Tyshenko, M.G., S. ElSaadany, T. Oraby, S. Darshan, W. Aspinall, R. Cooke, A. Catford, and D. Krewski (2011) Expert elicitation for the judgment of prion disease risk uncertainties. J Toxicol Environ Health A.; 74(2–4):261–285 Oraby,T., Tyshenko, M.G., Westphal, M., Darshan, S., Croteau, M., Aspinall, W., Elsaadany, S., Cashman, N. and Krewski, D. (2011) Using Expert Judgments to Improve Chronic Wasting Disease Risk Management in Canada. Journal of Toxicology and Environmental Health, in press. Volume 74, Issue 2–4, 2011 Special Issue: Prion Research in Perspective 2010 |
Nebraska | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
San Diego | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
BFIQ | Colson, A. Cooke, R.M., Lutter, Randall, (2016) How Does Breastfeeding Affect IQ? Applying the Classical Model of Structured Expert Judgment, Resources for the Future, RFF DP16–28 http://www.rff.org/research/publications/how-does-breastfeeding-affect-iq-applying-classical-model-structured-expert |
France | Abigail R. Colson, Itamar Megiddo, Gerardo Alvarez-Uria, Sumanth Gandra, Tim Bedford, Alec Morton, Roger M. Cooke, Ramanan Laxminarayan (ns). “Quantifying Uncertainty about Future Antimicrobial Resistance: Comparing Structured Expert Judgment and Statistical Forecasting Methods.” |
Italy | Abigail R. Colson, Itamar Megiddo, Gerardo Alvarez-Uria, Sumanth Gandra, Tim Bedford, Alec Morton, Roger M. Cooke, Ramanan Laxminarayan (ns). “Quantifying Uncertainty about Future Antimicrobial Resistance: Comparing Structured Expert Judgment and Statistical Forecasting Methods.” |
Spain | Abigail R. Colson, Itamar Megiddo, Gerardo Alvarez-Uria, Sumanth Gandra, Tim Bedford, Alec Morton, Roger M. Cooke, Ramanan Laxminarayan (ns). “Quantifying Uncertainty about Future Antimicrobial Resistance: Comparing Structured Expert Judgment and Statistical Forecasting Methods.” |
UK | Abigail R. Colson, Itamar Megiddo, Gerardo Alvarez-Uria, Sumanth Gandra, Tim Bedford, Alec Morton, Roger M. Cooke, Ramanan Laxminarayan (ns). “Quantifying Uncertainty about Future Antimicrobial Resistance: Comparing Structured Expert Judgment and Statistical Forecasting Methods.” |
Arkansas | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012. |
CoveringKids | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
dcpn_Fistula | Aspinall,W. Devleesschauwer, B. Cooke, R.M., Corrigan,T., Havelaar, A.H., Gibb, H., Torgerson, P., Kirk, M., Angulo, F., Lake, R., Speybroeck, N., and Hoffmann, S. (2015) World Health Organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: a structured expert elicitation. PLOS ONE,: January 19, 2016 https://doi.org/10.1371/journal.pone.0145839 |
Florida | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
Illinois | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
Obesity | Colson, Abigail R., R.M. Cooke, R. Laxminarayan. (2015) “Attributing Impact to a Charitable Foundation’s Programs with Structured Expert Judgment.” Working Paper. Center for Disease Dynamics, Economics & Policy |
Tobacco | Colson, Abigail R., R.M. Cooke, R. Laxminarayan. (2015) “Attributing Impact to a Charitable Foundation’s Programs with Structured Expert Judgment.” Working Paper. Center for Disease Dynamics, Economics & Policy |
Washington | Attribution study for Robert Wood Johnson Covering Kids & Families in Pennsylvania, Washington, Nebraska, Illinois, Arkansas, and Florida, conducted by Center for Disease Dynamics, Economics & Policy, 2012 |
cdc-roi | Colson, Abigail R., M.A. Cohen, S. Regmi, A. Nandi, R. Laxminarayan (2015) “Structured Expert Judgment for Informing the Return on Investment in Surveillance: The Case of Environmental Public Health Tracking.” Working Paper. Center for Disease Dynamics, Economics & Policy |
IQ-earn | Randall Lutter, Abigail Colson, and Roger Cooke (ns), (ns), “Effects of Increases in IQ in India on the Present Value of Lifetime Earnings |
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Marti, D., Mazzuchi, T.A., Cooke, R.M. (2021). Are Performance Weights Beneficial? Investigating the Random Expert Hypothesis. In: Hanea, A.M., Nane, G.F., Bedford, T., French, S. (eds) Expert Judgement in Risk and Decision Analysis. International Series in Operations Research & Management Science, vol 293. Springer, Cham. https://doi.org/10.1007/978-3-030-46474-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-46474-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-46473-8
Online ISBN: 978-3-030-46474-5
eBook Packages: Business and ManagementBusiness and Management (R0)