[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Analysis
  • Published:

Power failure: why small sample size undermines the reliability of neuroscience

Nature Reviews Neuroscience volume 14, pages 365–376 (2013)Cite this article

Subjects

An Erratum to this article was published on 15 April 2013

This article has been updated

Key Points

  • Low statistical power undermines the purpose of scientific research; it reduces the chance of detecting a true effect.

  • Perhaps less intuitively, low power also reduces the likelihood that a statistically significant result reflects a true effect.

  • Empirically, we estimate the median statistical power of studies in the neurosciences is between ∼8% and ∼31%.

  • We discuss the consequences of such low statistical power, which include overestimates of effect size and low reproducibility of results.

  • There are ethical dimensions to the problem of low power; unreliable research is inefficient and wasteful.

  • Improving reproducibility in neuroscience is a key priority and requires attention to well-established, but often ignored, methodological principles.

  • We discuss how problems associated with low power can be addressed by adopting current best-practice and make clear recommendations for how to achieve this.

Abstract

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Statistical power of a replication study.
Figure 2: Flow diagram of articles selected for inclusion.
Figure 3: Median power of studies included in neuroscience meta-analyses.
Figure 4: Positive predictive value as a function of the pre-study odds of association for different levels of statistical power.
Figure 5: The winner's curse: effect size inflation as a function of statistical power.

Similar content being viewed by others

Change history

  • 15 April 2013

    On page 2 of this article, the definition of R should have read: "R is the pre-study odds (that is, the odds that a probed effect is indeed non-null among the effects being probed)". This has been corrected in the online version.

References

  1. Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012).

    Article  Google Scholar 

  3. Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975).

    Article  Google Scholar 

  4. Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).

    Article  PubMed  Google Scholar 

  5. Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false.

    Article  PubMed  Google Scholar 

  6. Sullivan, P. F. Spurious genetic associations. Biol. Psychiatry 61, 1121–1126 (2007).

    Article  CAS  PubMed  Google Scholar 

  7. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).

    Article  CAS  PubMed  Google Scholar 

  8. Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011).

    Article  CAS  Google Scholar 

  9. Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119–120 (2009).

    Article  CAS  PubMed  Google Scholar 

  11. Sterne, J. A. & Davey Smith, G. Sifting the evidence — what's wrong with significance tests? BMJ 322, 226–231 (2001).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  12. Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).

    Article  PubMed  Google Scholar 

  13. Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008).

    Article  PubMed  Google Scholar 

  14. Tversky, A. & Kahneman, D. Belief in the law of small numbers. Psychol. Bull. 75, 105–110 (1971).

    Article  Google Scholar 

  15. Masicampo, E. J. & Lalande, D. R. A peculiar prevalence of p values just below .05. Q. J. Exp. Psychol. 65, 2271–2279 (2012).

    Article  CAS  Google Scholar 

  16. Carp, J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63, 289–300 (2012). This article reviews methods reporting and methodological choices across 241 recent fMRI studies and shows that there were nearly as many unique analytical pipelines as there were studies. In addition, many studies were underpowered to detect plausible effects.

    Article  PubMed  Google Scholar 

  17. Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Sterne, J. A. et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 343, d4002 (2011).

    Article  PubMed  Google Scholar 

  19. Joy-Gaba, J. A. & Nosek, B. A. The surprisingly limited malleability of implicit racial evaluations. Soc. Psychol. 41, 137–146 (2010).

    Article  Google Scholar 

  20. Schmidt, K. & Nosek, B. A. Implicit (and explicit) racial attitudes barely changed during Barack Obama's presidential campaign and early presidency. J. Exp. Soc. Psychol. 46, 308–314 (2010).

    Article  Google Scholar 

  21. Evangelou, E., Siontis, K. C., Pfeiffer, T. & Ioannidis, J. P. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J. Clin. Epidemiol. 65, 1274–1281 (2012).

    Article  PubMed  Google Scholar 

  22. Pereira, T. V. & Ioannidis, J. P. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J. Clin. Epidemiol. 64, 1060–1069 (2011).

    Article  PubMed  Google Scholar 

  23. Faul, F., Erdfelder, E., Lang, A. G. & Buchner, A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175–191 (2007).

    Article  PubMed  Google Scholar 

  24. Babbage, D. R. et al. Meta-analysis of facial affect recognition difficulties after traumatic brain injury. Neuropsychology 25, 277–285 (2011).

    Article  PubMed  Google Scholar 

  25. Bai, H. Meta-analysis of 5, 10-methylenetetrahydrofolate reductase gene poymorphism as a risk factor for ischemic cerebrovascular disease in a Chinese Han population. Neural Regen. Res. 6, 277–285 (2011).

    Google Scholar 

  26. Bjorkhem-Bergman, L., Asplund, A. B. & Lindh, J. D. Metformin for weight reduction in non-diabetic patients on antipsychotic drugs: a systematic review and meta-analysis. J. Psychopharmacol. 25, 299–305 (2011).

    Article  PubMed  Google Scholar 

  27. Bucossi, S. et al. Copper in Alzheimer's disease: a meta-analysis of serum, plasma, and cerebrospinal fluid studies. J. Alzheimers Dis. 24, 175–185 (2011).

    Article  CAS  PubMed  Google Scholar 

  28. Chamberlain, S. R. et al. Translational approaches to frontostriatal dysfunction in attention-deficit/hyperactivity disorder using a computerized neuropsychological battery. Biol. Psychiatry 69, 1192–1203 (2011).

    Article  PubMed  Google Scholar 

  29. Chang, W. P., Arfken, C. L., Sangal, M. P. & Boutros, N. N. Probing the relative contribution of the first and second responses to sensory gating indices: a meta-analysis. Psychophysiology 48, 980–992 (2011).

    Article  PubMed  Google Scholar 

  30. Chang, X. L. et al. Functional parkin promoter polymorphism in Parkinson's disease: new data and meta-analysis. J. Neurol. Sci. 302, 68–71 (2011).

    Article  CAS  PubMed  Google Scholar 

  31. Chen, C. et al. Allergy and risk of glioma: a meta-analysis. Eur. J. Neurol. 18, 387–395 (2011).

    Article  CAS  PubMed  Google Scholar 

  32. Chung, A. K. & Chua, S. E. Effects on prolongation of Bazett's corrected QT interval of seven second-generation antipsychotics in the treatment of schizophrenia: a meta-analysis. J. Psychopharmacol. 25, 646–666 (2011).

    Article  PubMed  Google Scholar 

  33. Domellof, E., Johansson, A. M. & Ronnqvist, L. Handedness in preterm born children: a systematic review and a meta-analysis. Neuropsychologia 49, 2299–2310 (2011).

    Article  PubMed  Google Scholar 

  34. Etminan, N., Vergouwen, M. D., Ilodigwe, D. & Macdonald, R. L. Effect of pharmaceutical treatment on vasospasm, delayed cerebral ischemia, and clinical outcome in patients with aneurysmal subarachnoid hemorrhage: a systematic review and meta-analysis. J. Cereb. Blood Flow Metab. 31, 1443–1451 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Feng, X. L. et al. Association of FK506 binding protein 5 (FKBP5) gene rs4713916 polymorphism with mood disorders: a meta-analysis. Acta Neuropsychiatr. 23, 12–19 (2011).

    Article  PubMed  Google Scholar 

  36. Green, M. J., Matheson, S. L., Shepherd, A., Weickert, C. S. & Carr, V. J. Brain-derived neurotrophic factor levels in schizophrenia: a systematic review with meta-analysis. Mol. Psychiatry 16, 960–972 (2011).

    Article  CAS  PubMed  Google Scholar 

  37. Han, X. M., Wang, C. H., Sima, X. & Liu, S. Y. Interleukin-6–74G/C polymorphism and the risk of Alzheimer's disease in Caucasians: a meta-analysis. Neurosci. Lett. 504, 4–8 (2011).

    Article  CAS  PubMed  Google Scholar 

  38. Hannestad, J., DellaGioia, N. & Bloch, M. The effect of antidepressant medication treatment on serum levels of inflammatory cytokines: a meta-analysis. Neuropsychopharmacology 36, 2452–2459 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Hua, Y., Zhao, H., Kong, Y. & Ye, M. Association between the MTHFR gene and Alzheimer's disease: a meta-analysis. Int. J. Neurosci. 121, 462–471 (2011).

    Article  CAS  PubMed  Google Scholar 

  40. Lindson, N. & Aveyard, P. An updated meta-analysis of nicotine preloading for smoking cessation: investigating mediators of the effect. Psychopharmacology 214, 579–592 (2011).

    Article  CAS  PubMed  Google Scholar 

  41. Liu, H. et al. Association of 5-HTT gene polymorphisms with migraine: a systematic review and meta-analysis. J. Neurol. Sci. 305, 57–66 (2011).

    Article  CAS  PubMed  Google Scholar 

  42. Liu, J. et al. PITX3 gene polymorphism is associated with Parkinson's disease in Chinese population. Brain Res. 1392, 116–120 (2011).

    Article  CAS  PubMed  Google Scholar 

  43. MacKillop, J. et al. Delayed reward discounting and addictive behavior: a meta-analysis. Psychopharmacology 216, 305–321 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Maneeton, N., Maneeton, B., Srisurapanont, M. & Martin, S. D. Bupropion for adults with attention-deficit hyperactivity disorder: meta-analysis of randomized, placebo-controlled trials. Psychiatry Clin. Neurosci. 65, 611–617 (2011).

    Article  PubMed  Google Scholar 

  45. Ohi, K. et al. The SIGMAR1 gene is associated with a risk of schizophrenia and activation of the prefrontal cortex. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 1309–1315 (2011).

    Article  CAS  PubMed  Google Scholar 

  46. Olabi, B. et al. Are there progressive brain changes in schizophrenia? A meta-analysis of structural magnetic resonance imaging studies. Biol. Psychiatry 70, 88–96 (2011).

    Article  PubMed  Google Scholar 

  47. Oldershaw, A. et al. The socio-emotional processing stream in Anorexia Nervosa. Neurosci. Biobehav. Rev. 35, 970–988 (2011).

    Article  CAS  PubMed  Google Scholar 

  48. Oliver, B. J., Kohli, E. & Kasper, L. H. Interferon therapy in relapsing-remitting multiple sclerosis: a systematic review and meta-analysis of the comparative trials. J. Neurol. Sci. 302, 96–105 (2011).

    Article  CAS  PubMed  Google Scholar 

  49. Peerbooms, O. L. et al. Meta-analysis of MTHFR gene variants in schizophrenia, bipolar disorder and unipolar depressive disorder: evidence for a common genetic vulnerability? Brain Behav. Immun. 25, 1530–1543 (2011).

    Article  CAS  PubMed  Google Scholar 

  50. Pizzagalli, D. A. Frontocingulate dysfunction in depression: toward biomarkers of treatment response. Neuropsychopharmacology 36, 183–206 (2011).

    Article  PubMed  Google Scholar 

  51. Rist, P. M., Diener, H. C., Kurth, T. & Schurks, M. Migraine, migraine aura, and cervical artery dissection: a systematic review and meta-analysis. Cephalalgia 31, 886–896 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  52. Sexton, C. E., Kalu, U. G., Filippini, N., Mackay, C. E. & Ebmeier, K. P. A meta-analysis of diffusion tensor imaging in mild cognitive impairment and Alzheimer's disease. Neurobiol. Aging 32, 2322.e5–2322.e18 (2011).

    Article  Google Scholar 

  53. Shum, D., Levin, H. & Chan, R. C. Prospective memory in patients with closed head injury: a review. Neuropsychologia 49, 2156–2165 (2011).

    Article  PubMed  Google Scholar 

  54. Sim, H. et al. Acupuncture for carpal tunnel syndrome: a systematic review of randomized controlled trials. J. Pain 12, 307–314 (2011).

    Article  PubMed  Google Scholar 

  55. Song, F. et al. Meta-analysis of plasma amyloid-β levels in Alzheimer's disease. J. Alzheimers Dis. 26, 365–375 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Sun, Q. L. et al. Correlation of E-selectin gene polymorphisms with risk of ischemic stroke A meta-analysis. Neural Regen. Res. 6, 1731–1735 (2011).

    CAS  Google Scholar 

  57. Tian, Y., Kang, L. G., Wang, H. Y. & Liu, Z. Y. Meta-analysis of transcranial magnetic stimulation to treat post-stroke dysfunction. Neural Regen. Res. 6, 1736–1741 (2011).

    Google Scholar 

  58. Trzesniak, C. et al. Adhesio interthalamica alterations in schizophrenia spectrum disorders: a systematic review and meta-analysis. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 877–886 (2011).

    Article  PubMed  Google Scholar 

  59. Veehof, M. M., Oskam, M. J., Schreurs, K. M. & Bohlmeijer, E. T. Acceptance-based interventions for the treatment of chronic pain: a systematic review and meta-analysis. Pain 152, 533–542 (2011).

    Article  PubMed  Google Scholar 

  60. Vergouwen, M. D., Etminan, N., Ilodigwe, D. & Macdonald, R. L. Lower incidence of cerebral infarction correlates with improved functional outcome after aneurysmal subarachnoid hemorrhage. J. Cereb. Blood Flow Metab. 31, 1545–1553 (2011).

    Article  PubMed  PubMed Central  Google Scholar 

  61. Vieta, E. et al. Effectiveness of psychotropic medications in the maintenance phase of bipolar disorder: a meta-analysis of randomized controlled trials. Int. J. Neuropsychopharmacol. 14, 1029–1049 (2011).

    Article  CAS  PubMed  Google Scholar 

  62. Wisdom, N. M., Callahan, J. L. & Hawkins, K. A. The effects of apolipoprotein E on non-impaired cognitive functioning: a meta-analysis. Neurobiol. Aging 32, 63–74 (2011).

    Article  CAS  PubMed  Google Scholar 

  63. Witteman, J., van Ijzendoorn, M. H., van de Velde, D., van Heuven, V. J. & Schiller, N. O. The nature of hemispheric specialization for linguistic and emotional prosodic perception: a meta-analysis of the lesion literature. Neuropsychologia 49, 3722–3738 (2011).

    Article  PubMed  Google Scholar 

  64. Woon, F. & Hedges, D. W. Gender does not moderate hippocampal volume deficits in adults with posttraumatic stress disorder: a meta-analysis. Hippocampus 21, 243–252 (2011).

    Article  PubMed  Google Scholar 

  65. Xuan, C. et al. No association between APOE ε 4 allele and multiple sclerosis susceptibility: a meta-analysis from 5472 cases and 4727 controls. J. Neurol. Sci. 308, 110–116 (2011).

    Article  CAS  PubMed  Google Scholar 

  66. Yang, W. M., Kong, F. Y., Liu, M. & Hao, Z. L. Systematic review of risk factors for progressive ischemic stroke. Neural Regen. Res. 6, 346–352 (2011).

    Google Scholar 

  67. Yang, Z., Li, W. J., Huang, T., Chen, J. M. & Zhang, X. Meta-analysis of Ginkgo biloba extract for the treatment of Alzheimer's disease. Neural Regen. Res. 6, 1125–1129 (2011).

    CAS  Google Scholar 

  68. Yuan, H. et al. Meta-analysis of tau genetic polymorphism and sporadic progressive supranuclear palsy susceptibility. Neural Regen. Res. 6, 353–359 (2011).

    Google Scholar 

  69. Zafar, S. N., Iqbal, A., Farez, M. F., Kamatkar, S. & de Moya, M. A. Intensive insulin therapy in brain injury: a meta-analysis. J. Neurotrauma 28, 1307–1317 (2011).

    Article  PubMed  Google Scholar 

  70. Zhang, Y. G. et al. The −1082G/A polymorphism in IL-10 gene is associated with risk of Alzheimer's disease: a meta-analysis. J. Neurol. Sci. 303, 133–138 (2011).

    Article  CAS  PubMed  Google Scholar 

  71. Zhu, Y., He, Z. Y. & Liu, H. N. Meta-analysis of the relationship between homocysteine, vitamin B(12), folate, and multiple sclerosis. J. Clin. Neurosci. 18, 933–938 (2011).

    Article  CAS  PubMed  Google Scholar 

  72. Ioannidis, J. P. & Trikalinos, T. A. An exploratory test for an excess of significant findings. Clin. Trials 4, 245–253 (2007). This study describes a test that evaluates whether there is an excess of significant findings in the published literature. The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies.

    Article  PubMed  Google Scholar 

  73. Ioannidis, J. P. Excess significance bias in the literature on brain volume abnormalities. Arch. Gen. Psychiatry 68, 773–780 (2011).

    Article  PubMed  Google Scholar 

  74. Pfeiffer, T., Bertram, L. & Ioannidis, J. P. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6, e18362 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  75. Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E. & Ioannidis, J. P. Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. J. Natl Cancer Inst. 104, 1867–1878 (2012).

    Article  CAS  PubMed  Google Scholar 

  76. Ioannidis, J. Clarifications on the application and interpretation of the test for excess significance and its extensions. J. Math. Psychol. (in the press).

  77. David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).

  78. Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  79. Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012).

    Article  PubMed  Google Scholar 

  80. Jonasson, Z. Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811–825 (2005).

    Article  PubMed  Google Scholar 

  81. Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).

    Article  PubMed  Google Scholar 

  82. Sena, E., van der Worp, H. B., Howells, D. & Macleod, M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 30, 433–439 (2007).

    Article  CAS  PubMed  Google Scholar 

  83. Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).

    Google Scholar 

  84. Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  85. Bassler, D., Montori, V. M., Briel, M., Glasziou, P. & Guyatt, G. Early stopping of randomized clinical trials for overt efficacy is problematic. J. Clin. Epidemiol. 61, 241–246 (2008).

    Article  PubMed  Google Scholar 

  86. Montori, V. M. et al. Randomized trials stopped early for benefit: a systematic review. JAMA 294, 2203–2209 (2005).

    Article  CAS  PubMed  Google Scholar 

  87. Berger, J. O. & Wolpert, R. L. The Likelihood Principle: A Review, Generalizations, and Statistical Implications (ed. Gupta, S. S.) (Institute of Mathematical Sciences, 1998).

    Google Scholar 

  88. Vesterinen, H. M. et al. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J. Cereb. Blood Flow Metab. 31, 1064–1072 (2011).

    Article  PubMed  Google Scholar 

  89. Smith, R. A., Levine, T. R., Lachlan, K. A. & Fediuk, T. A. The high cost of complexity in experimental design and data analysis: type I and type II error rates in multiway ANOVA. Hum. Comm. Res. 28, 515–530 (2002).

    Article  Google Scholar 

  90. Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334, 197 (2007).

    Article  CAS  PubMed  Google Scholar 

  91. Nosek, B. A. & Bar-Anan, Y. Scientific utopia: I. Opening scientific communication. Psychol. Inquiry 23, 217–243 (2012).

    Article  Google Scholar 

  92. Open-Science-Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657–660 (2012). This article describes the Reproducibility Project — an open, large-scale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science. This will allow the empirical rate of replication to be estimated.

  93. Simera, I. et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 8, 24 (2010).

    Article  PubMed  PubMed Central  Google Scholar 

  94. Ioannidis, J. P. The importance of potential studies that have not existed and registration of observational data sets. JAMA 308, 575–576 (2012).

    Article  CAS  PubMed  Google Scholar 

  95. Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H. & Ioannidis, J. P. Public availability of published research data in high-impact journals. PLoS ONE 6, e24357 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  96. Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149–155 (2009).

    Article  CAS  PubMed  Google Scholar 

  97. Ioannidis, J. P. & Khoury, M. J. Improving validation practices in “omics” research. Science 334, 1230–1232 (2011).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  98. Chambers, C. D. Registered Reports: A new publishing initiative at Cortex. Cortex 49, 609–610 (2013).

    Article  PubMed  Google Scholar 

  99. Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).

    Article  PubMed  Google Scholar 

  100. Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies. Eur. J. Hum. Genet. 18, 832–837 (2010).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  101. Ioannidis, J. P. & Trikalinos, T. A. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58, 543–549 (2005).

    Article  PubMed  Google Scholar 

  102. Ioannidis, J. Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645–654 (2012).

    Article  PubMed  Google Scholar 

  103. Zollner, S. & Pritchard, J. K. Overcoming the winner's curse: estimating penetrance parameters from case-control data. Am. J. Hum. Genet. 80, 605–615 (2007).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

M.R.M. and K.S.B. are members of the UK Centre for Tobacco Control Studies, a UK Public Health Research Centre of Excellence. Funding from British Heart Foundation, Cancer Research UK, Economic and Social Research Council, Medical Research Council and the UK National Institute for Health Research, under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged. The authors are grateful to G. Lewis for his helpful comments.

Author information

Authors and Affiliations

  1. School of Experimental Psychology, University of Bristol, Bristol, BS8 1TU, UK

    Katherine S. Button, Claire Mokrysz & Marcus R. Munafò

  2. School of Social and Community Medicine, University of Bristol, Bristol, BS8 2BN, UK

    Katherine S. Button

  3. Stanford University School of Medicine, Stanford, 94305, California, USA

    John P. A. Ioannidis

  4. Department of Psychology, University of Virginia, Charlottesville, 22904, Virginia, USA

    Brian A. Nosek

  5. Wellcome Trust Centre for Human Genetics, University of Oxford, Oxford, OX3 7BN, UK

    Jonathan Flint

  6. School of Physiology and Pharmacology, University of Bristol, Bristol, BS8 1TD, UK

    Emma S. J. Robinson

Authors
  1. Katherine S. Button
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. John P. A. Ioannidis
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Claire Mokrysz
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Brian A. Nosek
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Jonathan Flint
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Emma S. J. Robinson
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Marcus R. Munafò
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Marcus R. Munafò.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Related links

Related links

FURTHER INFORMATION

Marcus R. Munafò's homepage

CAMARADES

CONSORT

Dryad

EQUATOR Network

figshare

INDI

OASIS

OpenfMRI

OSF

PRISMA

The Dataverse Network Project

Rights and permissions

Reprints and permissions

About this article

Cite this article

Button, K., Ioannidis, J., Mokrysz, C. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14, 365–376 (2013). https://doi.org/10.1038/nrn3475

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrn3475

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing