Limits of empirical validation: a review of arguments with respect to social simulation
Pages 4069 - 4080
Abstract
Output comparison between simulation model and real world reference system is commonly regarded to be the acid test of model credibility. As sound as the comparison-based approach may seem, serious epistemological and methodological qualifications have been made concerning the foundations of the concept, its applicability, and its dependence from the chosen philosophical perspective. The article reviews and reassesses technical and philosophical arguments on the limits of empirical validation with respect to social simulation. The paper is intended to reposition empirical validation for social simulations that are theory-free and non-predictive. The proposed shift is inspired by the recent critical reassessment of significance tests in applied statistics. According to this shift, it is transparency which becomes paramount for the single social simulation project, whereas empirical validation on the macro level is crucial only after meta-analysis of rival simulation models has shown robust findings despite different sets of assumptions.
References
[1]
Ackoff, R. L. 1979. "Resurrecting the future of Operational Research". Journal of the Operational Research Society 30 (3): 189--199.
[2]
Ahrweiler, P., and N. Gilbert. 2005. "Cafe Nero: the evaluation of social simulation". Journal of Artificial Societies and Social Simulation 8 (4): 14.
[3]
Armstrong, J. S. 2007. "Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries". Internation Journal of Forecasting 23:335--336.
[4]
Arnold, E. 2013. "Simulation models of the evolution of cooperation as proofs of logical possibilities. How useful are they?". Ethics and Politics 2 (XV): 101--138.
[5]
Arnold, E. 2014. "What's wrong with social simulations?". The Monist 97 (3): 359--377.
[6]
Axelrod, R. 1984. The Evolution of Cooperation. Basic Books.
[7]
Bair, L., and A. Tolk. 2013. "Towards a unified theory of validation". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Pasupathy, S. Kim, A. Tolk, R. Hill, and M. Kuhl. Institute of Electrical and Electronics Engineers, Inc.
[8]
Bakan, D. 1966. "The test of significance in psychological research". Psychological Bulletin 66:423--437.
[9]
Balzer, W., C. Moulines, and J. Sneed. 1987. An Architectonic for Science. The Structuralist Program. Reidel.
[10]
Barlas, Y., and S. Carpenter. 1990. "Philosophical roots of model validation: Two paradigms". Systems Dynamics Review 6:48--166.
[11]
Barretto, M., L. Chwif, and R. J. Paul. 2000. "On simulation model complexity". In Proceedings of the 2000 Winter Simulation Conference, edited by J. Joines, R. Barton, K. Kang, and P. Fishwick, 449--455. Piscataway, New Jersey: Insitute of Electrical and Electronics Engineers.
[12]
Berkson, J. 1938. "Some significance of interpretation encountered in the application of the Chisquare test". Journal of the American Statistical Association 33:526--536.
[13]
Beven, K. 2002. "Towards a coherent philosophy for modelling the environment". In Proceedings of the Royal Society London, Volume 458, 1--20. Royal Society London.
[14]
Boring, E. 1919. "Mathematical vs. scientific significance". Psychological Bulletin 16:335--338.
[15]
Brewer, W. F., and B. L. Lambert. 2001. "The theory-ladenness of observation and the theory-ladenness of the rest of the scientific process". Philosophy of Science 68 (3): 176--186.
[16]
Byrne, D. 1997. "Simulation - a way forward?". Sociological Research Online 2 (2): 4.
[17]
Carley, K. 1996. "Validating computational models". Technical report, Carnegie Mellon University.
[18]
Carrier, M. 1994. The Completeness of Scientific Theories. Dordrecht: Kluwer.
[19]
Cohen, J. 1994. "The earth is round (p < 0.5)". American Psychologist 12:997--1003.
[20]
Fraassen, B. C. V. 1980. The Scientific Image. Oxford University Press.
[21]
Friedman, M. 1953. Essays in Positive Economics. Chicago: University of Chicago Press.
[22]
Goldspink, C. 2002. "Methodological implications of complex systems approaches to sociality: simulation as a foundation for knowledge". Journal of Artificial Societies and Social Simulation, JASSS 5 (1): 3.
[23]
Greenwald, A. 1975. "Consequences of predjudice agains the null hypothesis". Psychological Bulletin 82:1--20.
[24]
Hagen, R. 1997. "In praise of the null hypothesis test". American Psychologist 52:15--24.
[25]
Harding, S. 1976. Can Theories be Refuted? Essays on the Duhem-Quine Thesis. Reidel.
[26]
Helmer, O., and N. Rescher. 1959. "On the epistemology of the inexact sciences". Management Science 6:25--52.
[27]
Hemez, F. M. 2004. "The myth of science-based predictive modeling". In Foundations 2004 Workshop for Verification, Validation, and Accreditation in the 21st Century. Arizona State University, Tempe, Arizona: Los Alamos National Laboratory.
[28]
Hermann, C. F. 1967. "Validation problems in games and simulations with special reference to models of international politics". Behavioral Science 12:216--231.
[29]
Hofmann, M. 2002. "Validation: real world system knowledge, types of validity and credibility levels". In Proceedings of the 16. European simulation Multiconference. Darmstadt, DE: SCS Europe.
[30]
Hofmann, M. 2005. "On the complexity of parameter calibration in simulation models". Journal of Defense Modeling and Simulation 2 (4): 217--226.
[31]
Hofmann, M. 2013. "Simulation-based exploratory data generation and analysis (data farming): a critical reflection on its validity and methodology". The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology 10 (4): 381--393.
[32]
Hopcroft, J. E., and J. D. Ullman. 1979. Introduction to Automata Theory, Languages, and Computation. Addison Wesley.
[33]
Ioannidis, J. 2005. "Why most puplished research findings are false". PLoS Medicine 2 (8): e124.
[34]
James, W. 1890. Principles of Psychology, Volume 1. New York: Henry Hold and Company.
[35]
Klee, R. 1997. Introduction to the Philosophy of Sciende: Cutting Nature at Its Seams. Oxford University Press.
[36]
Kleindorfer, G. B., L. O'Neill, and R. Ganeshan. 1998. "Validation in simulation: various positions in the philosophy of science". Management Science 44 (8): 1087--1099.
[37]
Kliemt, H. 1996. Modelling and Simulation in the Social Sciences from the Philosophy of Science Point of View, Chapter Simulation and Rational Practice, 13--26. Dordrecht: Kluwer.
[38]
Konikov, L., and J. Bredehoeft. 1992. "Ground-water models cannot be validated". Adv. Water Resour. 15:75--83.
[39]
Lambdin, C. 2012. "Significance tests as sorcery: Science is empirical - significance tests are not". Theory and Psychology 22 (1): 67--90.
[40]
Landry, M., and M. Oral. 1993. "In search for a valid view of model validation for Operations Research". European Journal of Operational Research; Special Issue on Model Validation 66:161--167.
[41]
Maki, U. 2000. "Kinds of Assumptions and their Truth: Shaking an Untwisted F-Twist". Kyklos 53 (3): 317--336.
[42]
Maxwell, N. 1997, August. "A new conception of science". Physics World 13 (8): 17--18.
[43]
Mulaik, S., N. Raju, and R. Harshman. 1997. "There is a time and a place for significance testsng". In What if there were no significance tests?, edited by L. Harlowand, S. Mulaik, and J. Steiger, 65--115. Erlbaum.
[44]
Musgrave, A. 1981. "Unreal Assumptions in Economic Theory". Kyklos 34:377--387.
[45]
Naylor, T. H., and J. M. Finger. 1967. "Verification of computer simulation models". Management Science 14 (2): 92--102.
[46]
Nearing, M., G. Govers, and L. Norton. 99. "Variability in soil erosion data from replicated plots". Soil Science Society of America Journal 63:1829--1832.
[47]
Olsen, M., and M. Raunak. 2013. "A framework for simulation validation coverage". In Proceedings of the 2013 Winter Simulation Conference, edited by R. Pasupathy, S. Kim, A. Tolk, R. Hill, and M. Kuhl. Institute of Electrical and Electronics Engineers, Inc.
[48]
Oreskes, N., K. Shrader-Frechette, and K. Belitz. 1994, February. "Verification, validation, and confirmation of numerical models in the earth sciences". Science 263 (4): 641--646.
[49]
Pace, D. 2004. "Modeling and simulation verification and validation challenges". Johns Hopkins APL Technical Digest 25 (2): 163--172.
[50]
Paolo, E. D., J. Noble, and S. Bullock. 2000. "Simulation models as opaque thought experiments". In Articial Life VII: Proceedings of the Seventh International Conference on Articial Life, edited by M. A. Bedau, J. S. McCaskill, N. H. Packard, and S. Rasmussen, 497--506. Cambridge, MA: MIT: MIT Press.
[51]
Pedgren, C. D., R. E. Shannon, and R. P. Sadowski. 1995. Introduction to simulation using SIMAN. 2nd ed. McGraw-Hill.
[52]
Quine, W. 1977. Ontological Relativity. Columbia University Press.
[53]
Refsgaard, J., and H. Henriksen. 2004. "Modelling guidelines--terminology and guiding principles". Advances in Water Resources 27:71--82.
[54]
Richardson, K. A. 2003. "On the limits of bottom-Upp computer simulation: towards a nonlinear modeling culture". In Proceedings of the 36th Hawaii International Conference on System Science. Institute of Electrical and Electronics Engineers, Inc.
[55]
Ruphy, S. 2011. "Limits to modeling: balancing ambition and outcome in astrophysics and cosmology". Simulation and Gaming 42 (2): 177--194.
[56]
Salt, J. D. 1993. "Keynote Address: Simulation should be easy and fun!". In Proceedings of the 1993 Winter Simulation Conference, edited by G. W. Evans, M. Mollaghasemi, E. C. Russell, and W. E. Biles, 1--5. Piscataway, N. J.: Institute of Electrical and Electronics Engineers, Inc.
[57]
Schelling, T. 1971. "Dynamic models of segregation". The Journal of Mathematical Sociology 1 (2): 143--186.
[58]
Schmidt, F., and J. Hunter. 1997. "Eight common but false objections to the discontinuation of significance testing in the analysis of research datat". In What if there were no significance tests?, edited by L. L. Harlow, S. A. Mulaik, and J. H. Steiger, 37--64. Erlbaum.
[59]
Senn, S. 2001. "Two cheers for P-values?". Journal of Epidemiology and Biostatistics 6 (2): 193--204.
[60]
Thompson, B. 2007. "Effect sizes, confidence intervals, and confidence intervals for effect sizes". Psychology in the Schools 44 (5): 423--432.
[61]
Troitzsch, K. 2004. Networked Simulation and Simulated Networks, Chapter Validating simulation models, 265--270. SCS Publishing House.
[62]
Trucano, T., L. Swiler, T. Igusa, W. Oberkampf, and M. Pilch. 2006. "Calibration, validation, and sensitivity analysis: Whats what". Reliability Engineering and System Safety 91:1331--1357.
[63]
Tukey, J. 1991. "The philosophy of multiple comparison". Statistical Science 6:100--116.
[64]
v. Glasersfeld, E. 1997. The Construction of Knowledge: Contributions to Conceptual Semantics. Intersystems Publications.
[65]
van Horn, R. L. 1971. "Validation of Simulation Results". Management Science 17 (5): 247--258.
[66]
von Clausewitz, C. 1934 (1991). Vom Kriege (On War; 1832). F. Dmmlers Verlag.
[67]
Ward, S. C. 1989. "Arguments for constructively simple models". Journal of the Operational Research Society 40 (2): 141--153.
[68]
Wittgenstein, L. 1953. Philosophical Investigations. Oxford: Blackwell.
[69]
Zeigler, B., T. Kim, and H. Praehofer. 2000. Theory of Modeling and Simulation, 2nd ed. New York: Academic Press.
Recommendations
Limits to Modeling: Balancing Ambition and Outcome in Astrophysics and Cosmology
Computer simulations of complex physical objects and processes for which data are very sparse or inexistent have become a major tool of scientific investigation in astrophysics and cosmology. However, one must ask how these simulations acquire their ...
Philosophical foundations of computer simulation validation
Innovations in information technology (IT) have been responsible for the rapid growth of computer simulation, which in turn has encouraged IT innovation. However, there is a striking contrast between the technological sophistication that drives and is ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2015
4051 pages
ISBN:9781467397414
Sponsors
Publisher
IEEE Press
Publication History
Published: 06 December 2015
Check for updates
Qualifiers
- Research-article
Conference
WSC '15
Sponsor:
Acceptance Rates
WSC '15 Paper Acceptance Rate 202 of 296 submissions, 68%;
Overall Acceptance Rate 3,413 of 5,075 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 33Total Downloads
- Downloads (Last 12 months)3
- Downloads (Last 6 weeks)0
Reflects downloads up to 30 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in