Abstract
Controlled experiments, also called A/B tests or split tests, are used in software engineering to improve products by evaluating variants with user data. By parameterizing software systems, multivariate experiments can be performed automatically and in large scale, in this way, controlled experimentation is formulated as an optimization problem. Using genetic algorithms for automated experimentation requires repetitions to evaluate a variant, since the fitness function is noisy. We propose to combine genetic algorithms with bandit optimization to optimize where repetitions are evaluated, instead of uniform sampling. We setup a simulation environment that allows us to evaluate the solution, and see that it leads to increased fitness, population diversity, and rewards, compared to only genetic algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Fagerholm, F., Guinea, A.S., Mäenpää, H., Münch, J.: The RIGHT model for continuous experimentation. J. Syst. Softw. 123, 292–305 (2017)
Kohavi, R., Henne, R.M., Sommerfield, D.: Practical guide to controlled experiments on the web: listen to your customers not to the HiPPO. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 959–967 (2007)
Tamburrelli, G., Margara, A.: Towards automated A/B testing. In: Goues, C., Yoo, S. (eds.) SSBSE 2014. LNCS, vol. 8636, pp. 184–198. Springer, Cham (2014). doi:10.1007/978-3-319-09940-8_13
Melanie, M.: An Introduction to Genetic Algorithms. MIT Press, Cambridge (1999)
Burtini, G., Loeppky, J., Lawrence, R.: A Survey of Online Experiment Design with the Stochastic Multi-Armed Bandit. ArXiv e-prints (2015). arXiv:1510.00757v4 [stat.ML]
Eppstein, D.: Graph-theoretic solutions to computational geometry problems. In: Paul, C., Habib, M. (eds.) WG 2009. LNCS, vol. 5911, pp. 1–16. Springer, Heidelberg (2010). doi:10.1007/978-3-642-11409-0_1
Lipowski, A., Lipowska, D.: Roulette-wheel selection via stochastic acceptance. Phys. A: Stat. Mech. Appl. 391(6), 2193–2196 (2012)
Acknowledgment
This work was conducted within the Wallenberg Autonomous Systems and Software Program (WASP) (http://wasp-sweden.se).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ros, R., Bjarnason, E., Runeson, P. (2017). Automated Controlled Experimentation on Software by Evolutionary Bandit Optimization. In: Menzies, T., Petke, J. (eds) Search Based Software Engineering. SSBSE 2017. Lecture Notes in Computer Science(), vol 10452. Springer, Cham. https://doi.org/10.1007/978-3-319-66299-2_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-66299-2_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-66298-5
Online ISBN: 978-3-319-66299-2
eBook Packages: Computer ScienceComputer Science (R0)