[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v44y2020i1p3-50.html
   My bibliography  Save this article

Feasible Policy Evaluation by Design: A Randomized Synthetic Stepped-Wedge Trial of Mandated Disclosure in King County

Author

Listed:
  • Cassandra Handan-Nader
  • Daniel E. Ho
  • Becky Elias
Abstract
Evidence-based policy is limited by the perception that randomized controlled trials (RCTs) are expensive and infeasible. We argue that carefully tailored research design can overcome these challenges and enable more widespread randomized evaluations of policy implementation. We demonstrate how a stepped-wedge (randomized rollout) design that adapts synthetic control methods overcame substantial practical, administrative, political, and statistical constraints to evaluating King County’s new food safety rating system. The core RCT component of the evaluation came at little financial cost to the government, allowed the entire county to be treated, and resulted in no functional implementation delay. The case of restaurant sanitation grading has played a critical role in the scholarship on information disclosure, and our study provides the first evidence from a randomized trial of the causal effects of grading on health outcomes. We find that the grading system had no appreciable effects on foodborne illness, hospitalization, or food handling practices but that the system may have marginally increased public engagement by encouraging higher reporting.

Suggested Citation

  • Cassandra Handan-Nader & Daniel E. Ho & Becky Elias, 2020. "Feasible Policy Evaluation by Design: A Randomized Synthetic Stepped-Wedge Trial of Mandated Disclosure in King County," Evaluation Review, , vol. 44(1), pages 3-50, February.
    • Handle: RePEc:sae:evarev:v:44:y:2020:i:1:p:3-50
    DOI: 10.1177/0193841X20930852
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20930852
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20930852?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wong, M.R. & McKelvey, W. & Ito, K. & Schiff, C. & Jacobson, J.B. & Kass, D., 2015. "Impact of a letter-grade program on restaurant sanitary conditions and diner behavior in New York City," American Journal of Public Health, American Public Health Association, vol. 105(3), pages 81-87.
    2. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    3. Daniel E. Ho, 2015. "Randomizing ... What? A Field Experiment of Child Access Voting Laws," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 171(1), pages 150-170, March.
    4. Grambsch, Patricia, 2008. "Regression to the Mean, Murder Rates, and Shall-Issue Laws," The American Statistician, American Statistical Association, vol. 62(4), pages 289-295.
    5. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    6. repec:feb:artefa:0087 is not listed on IDEAS
    7. Esarey, Justin & Menger, Andrew, 2019. "Practical and Effective Approaches to Dealing With Clustered Data," Political Science Research and Methods, Cambridge University Press, vol. 7(3), pages 541-559, July.
    8. Levitt, Steven D. & List, John A., 2009. "Field experiments in economics: The past, the present, and the future," European Economic Review, Elsevier, vol. 53(1), pages 1-18, January.
    9. Jeffrey M Wooldridge, 2010. "Econometric Analysis of Cross Section and Panel Data," MIT Press Books, The MIT Press, edition 2, volume 1, number 0262232588, April.
    10. Alberto Abadie & Alexis Diamond & Jens Hainmueller, 2015. "Comparative Politics and the Synthetic Control Method," American Journal of Political Science, John Wiley & Sons, vol. 59(2), pages 495-510, February.
    11. Richard P. Nathan, 2008. "The role of random assignment in social policy research," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 27(3), pages 606-606.
    12. Ho, Daniel E. & Imai, Kosuke, 2006. "Randomization Inference With Natural Experiments: An Analysis of Ballot Effects in the 2003 California Recall Election," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 888-900, September.
    13. Martin Ravallion, 2012. "Fighting Poverty One Experiment at a Time: Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty : Review Essay," Journal of Economic Literature, American Economic Association, vol. 50(1), pages 103-114, March.
    14. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    15. Richard P. Nathan, 2008. "The role of random assignment in social policy research," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 27(2), pages 401-415.
    16. Daniel E. Ho & Zoe C. Ashwood & Cassandra Handan-Nader, 2019. "New Evidence on Information Disclosure through Restaurant Hygiene Grading," American Economic Journal: Economic Policy, American Economic Association, vol. 11(4), pages 404-428, November.
    17. Abadie, Alberto & Diamond, Alexis & Hainmueller, Jens, 2010. "Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 493-505.
    18. David Weil & Archon Fung & Mary Graham & Elena Fagotto, 2006. "The effectiveness of regulatory disclosure policies," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(1), pages 155-181.
    19. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    20. George Loewenstein & Cass R. Sunstein & Russell Golman, 2014. "Disclosure: Psychology Changes Everything," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 391-419, August.
    21. Matthew S. Johnson, 2020. "Regulation by Shaming: Deterrence Effects of Publicizing Violations of Workplace Safety and Health Laws," American Economic Review, American Economic Association, vol. 110(6), pages 1866-1904, June.
    22. Ginger Zhe Jin & Phillip Leslie, 2003. "The Effect of Information on Product Quality: Evidence from Restaurant Hygiene Grade Cards," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(2), pages 409-451.
    23. Helen Z. Margetts, 2011. "Experiments for Public Management Research," Public Management Review, Taylor & Francis Journals, vol. 13(2), pages 189-208, February.
    24. Rodrik, Dani, 2008. "The New Development Economics: We Shall Experiment, but How Shall We Learn?," Working Paper Series rwp08-055, Harvard University, John F. Kennedy School of Government.
    25. Clifford Winston, 2008. "The Efficacy of Information Policy: A Review of Archon Fung, Mary Graham, and David Weil's Full Disclosure: The Perils and Promise of Transparency," Journal of Economic Literature, American Economic Association, vol. 46(3), pages 704-717, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sviták, Jan & Tichem, Jan & Haasbeek, Stefan, 2021. "Price effects of search advertising restrictions," International Journal of Industrial Organization, Elsevier, vol. 77(C).
    2. Guido W. Imbens, 2022. "Causality in Econometrics: Choice vs Chance," Econometrica, Econometric Society, vol. 90(6), pages 2541-2566, November.
    3. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    4. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    5. Kovács, Balázs & Lehman, David W. & Carroll, Glenn R., 2020. "Grade inflation in restaurant hygiene inspections: Repeated interactions between inspectors and restaurateurs," Food Policy, Elsevier, vol. 97(C).
    6. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    7. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    8. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    9. Dennis Shen & Peng Ding & Jasjeet Sekhon & Bin Yu, 2022. "Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data," Papers 2207.14481, arXiv.org, revised Oct 2022.
    10. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    11. Sunil Mithas & Yanzhen Chen & Yatang Lin & Alysson De Oliveira Silveira, 2022. "On the causality and plausibility of treatment effects in operations management research," Production and Operations Management, Production and Operations Management Society, vol. 31(12), pages 4558-4571, December.
    12. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    13. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    14. Bautista, M. A. & González, F. & Martínez, L. R. & Muñoz, P. & Prem, M., 2020. "Chile’s Missing Students: Dictatorship, Higher Education and Social Mobility," Documentos de Trabajo 18163, Universidad del Rosario.
    15. Goryunov, Alexander & Ageshina, Elena & Lavrentev, Igor & Peretyatko, Polina, 2023. "Estimating the effect of Russia’s development policy in the Far Eastern region: The synthetic control approach," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 72, pages 58-72.
    16. Michał Marcin Kobierecki & Michał Pierzgalski, 2022. "Sports Mega-Events and Economic Growth: A Synthetic Control Approach," Journal of Sports Economics, , vol. 23(5), pages 567-597, June.
    17. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    18. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    19. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    20. Billy Ferguson & Brad Ross, 2020. "Assessing the Sensitivity of Synthetic Control Treatment Effect Estimates to Misspecification Error," Papers 2012.15367, arXiv.org, revised Feb 2021.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:44:y:2020:i:1:p:3-50. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.