More Web Proxy on the site http://driver.im/

Article

Small Solutions for Real-World Symbolic Regression Using Denoising Autoencoder Genetic Programming

Authors:

David Wittenberg,

Franz RothlaufAuthors Info & Claims

Genetic Programming: 26th European Conference, EuroGP 2023, Held as Part of EvoStar 2023, Brno, Czech Republic, April 12–14, 2023, Proceedings

Pages 101 - 116

https://doi.org/10.1007/978-3-031-29573-7_7

Published: 12 April 2023 Publication History

Abstract

Denoising Autoencoder Genetic Programming (DAE-GP) is a model-based evolutionary algorithm that uses denoising autoencoder long short-term memory networks as probabilistic model to replace the standard recombination and mutation operators of genetic programming (GP). In this paper, we use the DAE-GP to solve a set of nine standard real-world symbolic regression tasks. We compare the prediction quality of the DAE-GP to standard GP, geometric semantic GP (GSGP), and the gene-pool optimal mixing evolutionary algorithm for GP (GOMEA-GP), and find that the DAE-GP shows similar prediction quality using a much lower number of fitness evaluations than GSGP or GOMEA-GP. In addition, the DAE-GP consistently finds small solutions. The best candidate solutions of the DAE-GP are 69% smaller (median number of nodes) than the best candidate solutions found by standard GP. An analysis of the bias of the selection and variation step for both the DAE-GP and standard GP gives insight into why differences in solution size exist: the strong increase in solution size for standard GP is a result of both selection and variation bias. The results highlight that learning and sampling from a probabilistic model is a promising alternative to classic GP variation operators where the DAE-GP is able to generate small solutions for real-world symbolic regression tasks.

References

[1]

Brooks, T.F., Pope, D.S., Marcolini, M.A.: Airfoil self-noise and prediction, vol. 1218. In: National Aeronautics and Space Administration, Office of Management, Scientific and Technical Information Division (1989)

[2]

Chollet, F.: keras (2015). https://github.com/fchollet/keras

[3]

Cortez P, Cerdeira A, Almeida F, Matos T, and Reis J Modeling wine preferences by data mining from physicochemical properties Decis. Support Syst. 2009 47 4 547-553

Digital Library

[4]

Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

[5]

Fortin FA, De Rainville FM, Gardner MA, Parizeau M, and Gagńe C DEAP: evolutionary algorithms made easy J. Mach. Learn. Res. 2012 13 1 2171-2175

Digital Library

[6]

Gerritsma J, Onnink R, and Versluis A Geometry, resistance and stability of the delft systematic yacht hull series Int. Shipbuild. Prog. 1981 28 328 276-297

[7]

Harrison D Jr and Rubinfeld DL Hedonic housing prices and the demand for clean air J. Environ. Econ. Manag. 1978 5 1 81-102

[8]

Hasegawa, Y., Iba, H.: Estimation of Bayesian network for program generation. In: Proceedings of the Third Asian-Pacific Workshop on Genetic Programming, pp. 35–46. Hanoi, Vietnam (2006)

[9]

Hasegawa, Y., Iba, H.: Estimation of distribution algorithm based on probabilistic grammar with latent annotations. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2007), pp. 1043–1050. IEEE (2007).

[10]

Hasegawa Y and Iba H A Bayesian network approach to program generation IEEE Trans. Evol. Comput. 2008 12 6 750-764

Digital Library

[11]

Kim K, Shan Y, Nguyen XH, and McKay RI Probabilistic model building in genetic programming: a critical review Genet. Program Evolvable Mach. 2014 15 2 115-167

Digital Library

[12]

Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations. San Diego, CA, USA (2015)

[13]

Koza JR Genetic Programming: On the Programming of Computers by Means of Natural Selection 1992 Cambridge, London MIT Press

Digital Library

[14]

La Cava, W., Spector, L., Danai, K.: Epsilon-lexicase selection for regression. In: Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO 2016), pp. 741–748. Association for Computing Machinery, New York, NY, USA (2016).

Digital Library

[15]

Martins, J.F.B.S., Oliveira, L.O.V.B., Miranda, L.F., Casadei, F., Pappa, G.L.: Solving the exponential growth of symbolic regression trees in geometric semantic genetic programming. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018), pp. 1151–1158. Association for Computing Machinery, New York, NY, USA (2018).

Digital Library

[16]

de Melo, V.V., Vargas, D.V., Banzhaf, W.: Batch tournament selection for genetic programming: the quality of lexicase, the speed of tournament. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2019), pp. 994–1002. Association for Computing Machinery, New York, NY, USA (2019).

Digital Library

[17]

Moraglio A, Krawiec K, and Johnson CG Coello CAC, Cutello V, Deb K, Forrest S, Nicosia G, and Pavone M Geometric semantic genetic programming Parallel Problem Solving from Nature - PPSN XII 2012 Heidelberg Springer 21-31

Digital Library

[18]

Ni J, Drieberg RH, and Rockett PI The use of an analytic quotient operator in genetic programming IEEE Trans. Evol. Comput. 2013 17 1 146-152

Digital Library

[19]

Orzechowski, P., La Cava, W., Moore, J.H.: Where are we now? A large benchmark study of recent symbolic regression methods. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2018), pp. 1183–1190. Association for Computing Machinery, New York, NY, USA (2018).

Digital Library

[20]

Probst M and Rothlauf F Harmless overfitting: using denoising autoencoders in estimation of distribution algorithms J. Mach. Learn. Res. 2020 21 78 1-31 http://jmlr.org/papers/v21/16-543.html

[21]

Ratle A and Sebag M Collet P, Fonlupt C, Hao J-K, Lutton E, and Schoenauer M Avoiding the bloat with stochastic grammar-based genetic programming Artificial Evolution 2002 Heidelberg Springer 255-266

[22]

Rothlauf, F.: Design of Modern Heuristics: Principles and Application, 1st edn. Springer, Heidelberg (2011).

[23]

Salustowicz R and Schmidhuber J Probabilistic incremental program evolution Evol. Comput. 1997 5 2 123-141

Digital Library

[24]

Schmidt, M., Lipson, H.: Age-fitness pareto optimization. In: Riolo, R., McConaghy, T., Vladislavleva, E. (eds.) Genetic Programming Theory and Practice VIII, pp. 129–146. Springer, New York (2011).

[25]

Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video representations using LSTMs. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 843–852. ACM, Lille, France (2015).

Digital Library

[26]

Tsanas A, Little MA, McSharry PE, and Ramig LO Accurate telemonitoring of Parkinson’s disease progression by noninvasive speech tests IEEE Trans. Biomed. Eng. 2009 57 4 884-893

[27]

Tsanas A and Xifara A Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools Energy Build. 2012 49 560-567

[28]

Vaswani A et al. Attention is all you need Adv. Neural. Inf. Process. Syst. 2017 30 5998-6008

[29]

Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning (ICML 2008), pp. 1096–1103. ACM, Helsinki, Finland (2008).

Digital Library

[30]

Virgolin M, Alderliesten T, Witteveen C, and Bosman PAN Improving model-based genetic programming for symbolic regression of small expressions Evol. Comput. 2021 29 2 211-237

[31]

Wittenberg, D.: Using denoising autoencoder genetic programming to control exploration and exploitation in search. In: Medvet, E., Pappa, G., Xue, B. (eds.) Genetic Programming (EuroGP 2022). LNCS, vol. 13223, pp. 102–117. Springer, Cham (2022).

Digital Library

[32]

Wittenberg, D., Rothlauf, F.: Denoising autoencoder genetic programming for real-world symbolic regression. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO 2022), pp. 612–614. Association for Computing Machinery, New York, NY, USA (2022).

Digital Library

[33]

Wittenberg, D., Rothlauf, F., Schweim, D.: DAE-GP: denoising autoencoder LSTM networks as probabilistic models in estimation of distribution genetic programming. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference (GECCO 2020), pp. 1037–1045. ACM, New York, NY, USA (2020).

Digital Library

[34]

Wong, P.K., Lo, L.Y., Wong, M.L., Leung, K.S.: Grammar-based genetic programming with Bayesian network. In: IEEE Congress on Evolutionary Computation (CEC 2014), pp. 739–746. IEEE, Beijing, China (2014)

[35]

Wong, P.K., Lo, L.Y., Wong, M.L., Leung, K.S.: Grammar-based genetic programming with dependence learning and Bayesian network classifier. In: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2014), pp. 959–966. ACM, Vancouver, Canada (2014).

Digital Library

[36]

Yanai, K., Iba, H.: Estimation of distribution programming based on Bayesian network. In: IEEE Congress on Evolutionary Computation (CEC 2003), pp. 1618–1625. IEEE, Canberra, Australia (2003).

[37]

Yeh IC Modeling of strength of high-performance concrete using artificial neural networks Cem. Concr. Res. 1998 28 12 1797-1808

Cited By

Reiter JSchweim DWittenberg DSilva SPaquete L(2023)Pretraining Reduces Runtime in Denoising Autoencoder Genetic Programming by an Order of MagnitudeProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3596332(2382-2385)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583133.3596332
Wittenberg DRothlauf FGagné C(2023)Denoising autoencoder genetic programming: strategies to control exploration and exploitation in searchGenetic Programming and Evolvable Machines10.1007/s10710-023-09462-224:2Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1007/s10710-023-09462-2

Recommendations

Denoising autoencoder genetic programming for real-world symbolic regression
GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Denoising Autoencoder Genetic Programming (DAE-GP) is a novel neural-network based estimation of distribution genetic programming algorithm that uses denoising autoencoder long short-term memory networks as probabilistic model to replace the standard ...
Genetic programming performance prediction and its application for symbolic regression problems
Highlights
- Theoretical analysis of GP performance prediction.
- Suggestion of upper bounds ...
Abstract
Predicting the performance of Genetic Programming (GP) helps us identify whether it is an appropriate approach to solve the problem at hand. However, previous studies show that measuring the difficulty of a problem for GP and ...
A Comparison of three evolutionary strategies for multiobjective genetic programming

We report what we believe to be the first comparative study of multi-objective genetic programming (GP) algorithms on benchmark symbolic regression and machine learning problems. We compare the Strength Pareto Evolutionary Algorithm (SPEA2), the Non-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Genetic Programming: 26th European Conference, EuroGP 2023, Held as Part of EvoStar 2023, Brno, Czech Republic, April 12–14, 2023, Proceedings

Apr 2023

365 pages

ISBN:978-3-031-29572-0

DOI:10.1007/978-3-031-29573-7

Editors:
Gisele Pappa
Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil
,
Mario Giacobini
Università degli studi di Torino, Turin, Italy
,
Zdenek Vasicek
Brno University of Technology, Brno, Czech Republic

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 12 April 2023

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Reiter JSchweim DWittenberg DSilva SPaquete L(2023)Pretraining Reduces Runtime in Denoising Autoencoder Genetic Programming by an Order of MagnitudeProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3596332(2382-2385)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583133.3596332
Wittenberg DRothlauf FGagné C(2023)Denoising autoencoder genetic programming: strategies to control exploration and exploitation in searchGenetic Programming and Evolvable Machines10.1007/s10710-023-09462-224:2Online publication date: 8-Nov-2023
https://dl.acm.org/doi/10.1007/s10710-023-09462-2

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents