[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
IDEAS home Printed from https://ideas.repec.org/p/cwl/cwldpp/1961.html
   My bibliography  Save this paper

A Simple Adjustment for Bandwidth Snooping

Author

Listed:
Abstract
Kernel-based estimators are often evaluated at multiple bandwidths as a form of sensitivity analysis. However, if in the reported results, a researcher selects the bandwidth based on this analysis, the associated confidence intervals may not have correct coverage, even if the estimator is unbiased. This paper proposes a simple adjustment that gives correct coverage in such situations: replace the Normal quantile with a critical value that depends only on the kernel and ratio of the maximum and minimum bandwidths the researcher has entertained. We tabulate these critical values and quantify the loss in coverage for conventional confidence intervals. For a range of relevant cases, a conventional 95% confidence interval has coverage between 70% and 90%, and our adjustment amounts to replacing the conventional critical value 1.96 with a number between 2.2 and 2.8. A Monte Carlo study confirms that our approach gives accurate coverage in finite samples. We illustrate our approach with two empirical applications.

Suggested Citation

  • Timothy B. Armstrong & Michal Kolesar, 2014. "A Simple Adjustment for Bandwidth Snooping," Cowles Foundation Discussion Papers 1961, Cowles Foundation for Research in Economics, Yale University.
    • Handle: RePEc:cwl:cwldpp:1961
    Note: Includes Supplemental Material
    as

    Download full text from publisher

    File URL: https://cowles.yale.edu/sites/default/files/files/pub/d19/d1961-a.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2009. "Dealing with limited overlap in estimation of average treatment effects," Biometrika, Biometrika Trust, vol. 96(1), pages 187-199.
    2. Johnston, Gordon J., 1982. "Probabilities of maximal deviations for nonparametric regression function estimates," Journal of Multivariate Analysis, Elsevier, vol. 12(3), pages 402-414, September.
    3. James J. Heckman & Sergio Urzua & Edward Vytlacil, 2006. "Understanding Instrumental Variables in Models with Essential Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 88(3), pages 389-432, August.
    4. Shao, Q. M., 1995. "Strong Approximation Theorems for Independent Random Variables and Their Applications," Journal of Multivariate Analysis, Elsevier, vol. 52(1), pages 107-130, January.
    5. David Card & David S. Lee & Zhuan Pei & Andrea Weber, 2015. "Inference on Causal Effects in a Generalized Regression Kink Design," Econometrica, Econometric Society, vol. 83, pages 2453-2483, November.
    6. Sebastian Galiani & Paul Gertler & Ernesto Schargrodsky, 2005. "Water for Life: The Impact of the Privatization of Water Services on Child Mortality," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 83-120, February.
    7. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    8. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    9. Lemieux, Thomas & Milligan, Kevin, 2008. "Incentive effects of social assistance: A regression discontinuity approach," Journal of Econometrics, Elsevier, vol. 142(2), pages 807-828, February.
    10. Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.
    11. James J. Heckman & Edward Vytlacil, 2005. "Structural Equations, Treatment Effects, and Econometric Policy Evaluation," Econometrica, Econometric Society, vol. 73(3), pages 669-738, May.
    12. Chamberlain, Gary, 1986. "Asymptotic efficiency in semi-parametric models with censoring," Journal of Econometrics, Elsevier, vol. 32(2), pages 189-218, July.
    13. Andrew Gelman & Guido Imbens, 2019. "Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(3), pages 447-456, July.
    14. Armstrong, Timothy B. & Chan, Hock Peng, 2016. "Multiscale adaptive inference on conditional moment inequalities," Journal of Econometrics, Elsevier, vol. 194(1), pages 24-43.
    15. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    16. David Card & Carlos Dobkin & Nicole Maestas, 2009. "Does Medicare Save Lives?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(2), pages 597-636.
    17. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2006. "Moving the Goalposts: Addressing Limited Overlap in the Estimation of Average Treatment Effects by Changing the Estimand," NBER Technical Working Papers 0330, National Bureau of Economic Research, Inc.
    18. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    19. DiNardo, John & Lee, David S., 2011. "Program Evaluation and Research Designs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 4, chapter 5, pages 463-536, Elsevier.
    20. Donald W. K. Andrews & Marcia M. A. Schafgans, 1998. "Semiparametric Estimation of the Intercept of a Sample Selection Model," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(3), pages 497-517.
    21. Jens Ludwig & Douglas L. Miller, 2007. "Does Head Start Improve Children's Life Chances? Evidence from a Regression Discontinuity Design," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(1), pages 159-208.
    22. Shakeeb Khan & Elie Tamer, 2010. "Irregular Identification, Support Conditions, and Inverse Weight Estimation," Econometrica, Econometric Society, vol. 78(6), pages 2021-2042, November.
    23. Victor Chernozhukov & Denis Chetverikov & Kengo Kato, 2012. "Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors," Papers 1212.6906, arXiv.org, revised Jan 2018.
    24. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    25. Frolich, Markus, 2007. "Nonparametric IV estimation of local average treatment effects with covariates," Journal of Econometrics, Elsevier, vol. 139(1), pages 35-75, July.
    26. Martha J. Bailey & Andrew Goodman-Bacon, 2015. "The War on Poverty's Experiment in Public Medicine: Community Health Centers and the Mortality of Older Americans," American Economic Review, American Economic Association, vol. 105(3), pages 1067-1104, March.
    27. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    28. Susanne M Schennach, 2020. "A Bias Bound Approach to Non-parametric Inference," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(5), pages 2439-2472.
    29. Timothy B. Armstrong, 2014. "Adaptive Testing on a Regression Function at a Point," Cowles Foundation Discussion Papers 1957, Cowles Foundation for Research in Economics, Yale University, revised Oct 2014.
    30. Heckman, James J, 1990. "Varieties of Selection Bias," American Economic Review, American Economic Association, vol. 80(2), pages 313-318, May.
    31. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    32. Wilbert van der Klaauw, 2002. "Estimating the Effect of Financial Aid Offers on College Enrollment: A Regression-Discontinuity Approach," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 43(4), pages 1249-1287, November.
    33. Lee, David S., 2008. "Randomized experiments from non-random selection in U.S. House elections," Journal of Econometrics, Elsevier, vol. 142(2), pages 675-697, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kato, Kengo & Sasaki, Yuya, 2019. "Uniform confidence bands for nonparametric errors-in-variables regression," Journal of Econometrics, Elsevier, vol. 213(2), pages 516-555.
    2. Susanne M Schennach, 2020. "A Bias Bound Approach to Non-parametric Inference," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(5), pages 2439-2472.
    3. Kato, Kengo & Sasaki, Yuya, 2018. "Uniform confidence bands in deconvolution with unknown error distribution," Journal of Econometrics, Elsevier, vol. 207(1), pages 129-161.
    4. Byunghoon Kang, 2019. "Inference in Nonparametric Series Estimation with Specification Searches for the Number of Series Terms," Papers 1909.12162, arXiv.org, revised Feb 2020.
    5. Xu, Ke-Li, 2017. "Regression discontinuity with categorical outcomes," Journal of Econometrics, Elsevier, vol. 201(1), pages 1-18.
    6. Chen, Heng & Fan, Yanqin, 2019. "Identification and wavelet estimation of weighted ATE under discontinuous and kink incentive assignment mechanisms," Journal of Econometrics, Elsevier, vol. 212(2), pages 476-502.
    7. Xu, Ke-Li, 2018. "A semi-nonparametric estimator of regression discontinuity design with discrete duration outcomes," Journal of Econometrics, Elsevier, vol. 206(1), pages 258-278.
    8. Jun Ma & Zhengfei Yu, 2020. "Empirical Likelihood Covariate Adjustment for Regression Discontinuity Designs," Papers 2008.09263, arXiv.org, revised May 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Mauricio Villamizar‐Villegas & Freddy A. Pinzon‐Puerto & Maria Alejandra Ruiz‐Sanchez, 2022. "A comprehensive history of regression discontinuity designs: An empirical survey of the last 60 years," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1130-1178, September.
    3. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    4. David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 281-355, June.
    5. Blaise Melly & Rafael Lalive, 2020. "Estimation, Inference, and Interpretation in the Regression Discontinuity Design," Diskussionsschriften dp2016, Universitaet Bern, Departement Volkswirtschaft.
    6. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    7. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    8. Jin-young Choi & Myoung-jae Lee, 2017. "Regression discontinuity: review with extensions," Statistical Papers, Springer, vol. 58(4), pages 1217-1246, December.
    9. Guido Imbens & Stefan Wager, 2019. "Optimized Regression Discontinuity Designs," The Review of Economics and Statistics, MIT Press, vol. 101(2), pages 264-278, May.
    10. Christelis, Dimitris & Georgarakos, Dimitris & Sanz-de-Galdeano, Anna, 2020. "The impact of health insurance on stockholding: A regression discontinuity approach," Journal of Health Economics, Elsevier, vol. 69(C).
    11. Ari Hyytinen & Jaakko Meriläinen & Tuukka Saarimaa & Otto Toivanen & Janne Tukiainen, 2018. "When does regression discontinuity design work? Evidence from random election outcomes," Quantitative Economics, Econometric Society, vol. 9(2), pages 1019-1051, July.
    12. Markus Frölich & Martin Huber, 2019. "Including Covariates in the Regression Discontinuity Design," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(4), pages 736-748, October.
    13. Timothy B. Armstrong & Michal Kolesár, 2021. "Finite‐Sample Optimal Estimation and Inference on Average Treatment Effects Under Unconfoundedness," Econometrica, Econometric Society, vol. 89(3), pages 1141-1177, May.
    14. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    15. Likai Chen & Georg Keilbar & Liangjun Su & Weining Wang, 2023. "Inference on many jumps in nonparametric panel regression models," Papers 2312.01162, arXiv.org, revised Aug 2024.
    16. Steven F. Koch & Jeffrey S. Racine, 2016. "Healthcare facility choice and user fee abolition: regression discontinuity in a multinomial choice setting," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 927-950, October.
    17. Yang He & Otávio Bartalotti, 2020. "Wild bootstrap for fuzzy regression discontinuity designs: obtaining robust bias-corrected confidence intervals," The Econometrics Journal, Royal Economic Society, vol. 23(2), pages 211-231.
    18. Bartalotti, Otávio C. & Calhoun, Gray & He, Yang, 2016. "Bootstrap Confidence Intervals for Sharp Regression Discontinuity Designs with the Uniform Kernel," Staff General Research Papers Archive 3394, Iowa State University, Department of Economics.
    19. Porter, Jack & Yu, Ping, 2015. "Regression discontinuity designs with unknown discontinuity points: Testing and estimation," Journal of Econometrics, Elsevier, vol. 189(1), pages 132-147.
    20. van der Klaauw, Bas, 2014. "From micro data to causality: Forty years of empirical labor economics," Labour Economics, Elsevier, vol. 30(C), pages 88-97.

    More about this item

    Keywords

    Nonparametric estimation; Multiple testing; Regression discontinuity;
    All these keywords.

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cwl:cwldpp:1961. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Brittany Ladd (email available below). General contact details of provider: https://edirc.repec.org/data/cowleus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.