[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.5555/3666122.3667522guideproceedingsArticle/Chapter ViewAbstractPublication PagesnipsConference Proceedingsconference-collections
research-article

CLIP-OGD: an experimental design for adaptive neyman allocation in sequential experiments

Published: 30 May 2024 Publication History

Abstract

From clinical development of cancer therapies to investigations into partisan bias, adaptive sequential designs have become increasingly popular method for causal inference, as they offer the possibility of improved precision over their non-adaptive counterparts. However, even in simple settings (e.g. two treatments) the extent to which adaptive designs can improve precision is not sufficiently well understood. In this work, we study the problem of Adaptive Neyman Allocation in a design-based potential outcomes framework, where the experimenter seeks to construct an adaptive design which is nearly as efficient as the optimal (but infeasible) non-adaptive Neyman design, which has access to all potential outcomes. Motivated by connections to online optimization, we propose Neyman Ratio and Neyman Regret as two (equivalent) performance measures of adaptive designs for this problem. We present CLIP-OGD, an adaptive design which achieves Õ(√T) expected Neyman regret and thereby recovers the optimal Neyman variance in large samples. Finally, we construct a conservative variance estimator which facilitates the development of asymptotically valid confidence intervals. To complement our theoretical results, we conduct simulations using data from a microeconomic experiment.

References

[1]
P.M. Aronow and Cyrus Samii. Conservative variance estimation for sampling designs with zero pairwise inclusion probabilities. Survey Methodology, 39(1):231-241, 2013.
[2]
P.M. Aronow and Cyrus Samii. Estimating average causal effects under general interference. Annals of Applied Statistics, 11(4):1912-1947, 2017.
[3]
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, may 2002a. ISSN 0885-6125.
[4]
Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002b.
[5]
AD Barker, CC Sigman, GJ Kelloff, NM Hylton, DA Berry, and LJ Esserman. I-spy 2: An adaptive breast cancer trial design in the setting of neoadjuvant chemotherapy. Clinical Pharmacology & Therapeutics, 86(1):97-100, 2009.
[6]
Matthew Blackwell, Nicole E. Pashley, and Dominic Valentino. Batch adaptive designs to improve efficiency in social science experiments. Working paper, Harvard University, 2022. URL https://www.mattblackwell.org/files/papers/batch_adaptive.pdf.
[7]
Jack Bowden and Lorenzo Trippa. Unbiased estimation for response adaptive clinical trials. Statistical methods in medical research, 26, 08 2015.
[8]
Giuseppe Burtini, Jason Loeppky, and Ramon Lawrence. A survey of online experiment design with the stochastic multi-armed bandit. arXiv preprint arXiv:1510.00757, 2015.
[9]
Yong Cai and Ahnaf Rafi. On the performance of the neyman allocation with small pilots. arXiv:2206.04643, 2022.
[10]
Jeffrey R. Eisele. The doubly adaptive biased coin design for sequential clinical trials. Journal of Statistical Planning and Inference, 38(2):249-261, 1994. ISSN 0378-3758.
[11]
National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical principles and guidelines for the commission for the protection of human subjects of biomedical and behavioral research. Technical report, US Department of Health, Education, and Welfare, 1978.
[12]
David A. Freedman. On regression adjustments to experimental data. Advances in Applied Mathematics, 40:180-193, 2008.
[13]
Matthew Groh and David McKenzie. Macroinsurance for microenterprises: A randomized experiment in post-revolution Egypt. Journal of Development Economics, 118:13-25, 2016.
[14]
Vitor Hadad, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. PNAS, 118(15), 2021.
[15]
Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics, 29(1):96-108, 2011.
[16]
Dae Woong Ham, Iavor Bojinov, Michael Lindon, and Martin Tingley. Design-based confidence sequences for anytime-valid causal inference. arXiv:2210.08639, 2022.
[17]
Christopher Harshaw, Joel A. Middleton, and Fredrik Sävje. Optimized variance estimation under interference and complex experimental designs. arXiv:2112.01709, 2021.
[18]
Christopher Harshaw, Fredrik Sävje, and Yitan Wang. A design-based riesz representation framework for randomized experiments. arXiv:2210.08698, 2022.
[19]
Lakhbir S. Hayre and Bruce W. Turnbull. Estimation of the odds ratio in the two-armed bandit problem. Biometrika, 68(3):661-668, 1981.
[20]
Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2(3-4):157-325, 2016. ISSN 2167-3888.
[21]
D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663-685, 1952.
[22]
Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055-1080, 2021.
[23]
Feifang Hu and William F. Rosenberger. Optimality, variability, power: Evaluating response-adaptive randomization procedures for treatment comparisons. Journal of the American Statistical Association, 98(463):671-678, 2003.
[24]
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[25]
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, 2020.
[26]
Michael P. Leung. Causal inference under approximate neighborhood interference. Econometrica, 90(1):267-293, 2022.
[27]
Winston Lin. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique. Annals of Applied Statistics, 7(1):295-318, 2013.
[28]
R. Narain. On sampling without replacement with varying probabilities. Journal of the Indian Society of Agricultural Statistics, 3:169—175, 1951.
[29]
Jerzy Neyman. On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Statistical Science, 5(4):465-472, 1923. Reprinted in 1990.
[30]
Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4): 558-625, 1934. ISSN 09528385.
[31]
Molly Offer-Westort, Alexander Coppock, and Donald P. Green. Adaptive experimental design: Prospects and applications in political science. American Journal of Political Science, 65(4): 826-844, 2021.
[32]
Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527 - 535, 1952.
[33]
W Rosenberger, N Stallard, Anastasia Ivanova, C Harper, and M Ricks. Optimal adaptive designs for binary response trials. Biometrics, 57:909-13, 10 2001.
[34]
Donald B. Rubin. Comment: Randomization analysis of experimental data. Journal of the American Statistical Association, 75(371):591, 1980.
[35]
Fredrik Sävje, P. M. Aronow, and Michael G. Hudgens. Average treatment effects in the presence of unknown interference. Annals of Statistics, 49(2):673-701, 2021.
[36]
Eric M. Schwartz, Eric T. Bradlow, and Peter S. Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4):500-522, 2017.
[37]
H. Solomon and S. Zacks. Optimal design of sampling from finite populations: A critical review and indication of new research areas. Journal of the American Statistical Association, 65(330): 653-677, 1970.
[38]
A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2): 117-186, 1945.
[39]
Yanxun Xu, Lorenzo Trippa, Peter Müller, and Yuan Ji. Subgroup-based adaptive (suba) designs for multi-arm biomarker trials. Statistics in Biosciences, 8, 02 2014.
[40]
Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. In Advances in Neural Information Processing Systems, volume 33, pages 9818-9829. Curran Associates, Inc., 2020.
[41]
Kelly Zhang, Lucas Janson, and Susan Murphy. Statistical inference with m-estimators on adaptively collected data. In Advances in Neural Information Processing Systems, volume 34, pages 7460-7471. Curran Associates, Inc., 2021.
[42]
Lanju Zhang and William F. Rosenberger. Response-adaptive randomization for clinical trials with continuous outcomes. Biometrics, 62(2):562-569, 2006.

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings
NIPS '23: Proceedings of the 37th International Conference on Neural Information Processing Systems
December 2023
80772 pages

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 30 May 2024

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 27 Dec 2024

Other Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media