[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3378184.3378187acmotherconferencesArticle/Chapter ViewAbstractPublication PagesappisConference Proceedingsconference-collections
research-article

Effective Mutation and Recombination for Evolving Convolutional Networks

Published: 17 February 2020 Publication History

Abstract

The major part of the success that we have had in deep learning is attributed to the proper design of the architecture of the network model. With all the hardware resources and mathematical foundation at hand, it is the clever assembling of the pieces that define the performance of that deep learning model. Almost all of the research in this field till date use the models that are designed by a human researcher using their past experiences or applying the iterative process of adding components to the model to see which one performs the best. As we strive towards the general Artificial Intelligence, this process of manually tuning the network to make it work on a specific task does not add up well for the cause. We are concerned with automating the task of network architecture design where the learning algorithm tries to discover the best performing model on itself. To do this, we employ an evolving algorithm in addition to the gradient descent to evolve and train the model. Specifically, we use a form of genetic algorithm with mutation and recombination operators to constantly change the architecture, while the commonly used gradient descent is used as a learning algorithm for each of the genetically procreated models. We propose a series of mutation operators and a method of recombination called Highest Varying k-Features Recombination(HVk-FR) to evolve the CNN models. Results show using our method of recombination on top of mutation yields the best accuracy.

References

[1]
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. 2016. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167 (2016).
[2]
Justin Bayer, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. 2009. Evolving memory cell structures for sequence learning. In International Conference on Artificial Neural Networks. Springer, 755--764.
[3]
Petet J Bentley and SP Kumar. [n. d.]. The ways to grow designs: A comparison of embryogenies for an evolutionary design problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 99). 35--43.
[4]
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. 2018. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In Advances in Neural Information Processing Systems. 5027--5038.
[5]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[6]
Gregory S Hornby and Jordan B Pollack. 2002. Creating high-level components with a generative representation for body-brain evolution. Artificial life 8, 3 (2002), 223--246.
[7]
Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In International Conference on Machine Learning. 2342--2350.
[8]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.
[9]
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. 2018a. ES is more than just a traditional finite-difference approximator. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 450--457.
[10]
Joel Lehman, Jay Chen, Jeff Clune, and Kenneth O Stanley. 2018b. Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 117--124.
[11]
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, et al. 2019. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing. Elsevier, 293--312.
[12]
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. 2017. Large-scale evolution of image classifiers. arXiv preprint arXiv:1703.01041 (2017).
[13]
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 (2017).
[14]
Shreyas Saxena and Jakob Verbeek. 2016. Convolutional neural fabrics. In Advances in Neural Information Processing Systems. 4053--4061.
[15]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[16]
Kenneth O Stanley. 2007. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines 8, 2 (2007), 131--162.
[17]
Kenneth O Stanley, David B D'Ambrosio, and Jason Gauci. 2009. A hypercube-based encoding for evolving large-scale neural networks. Artificial life 15, 2 (2009), 185--212.
[18]
Kenneth O Stanley and Risto Miikkulainen. 2002. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99--127.
[19]
Kenneth O Stanley and Risto Miikkulainen. 2003. A taxonomy for artificial embryogeny. Artificial Life 9, 2 (2003), 93--130.
[20]
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O Stanley, and Jeff Clune. 2017. Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567 (2017).
[21]
Yanan Sun, Gary G Yen, and Zhang Yi. 2018. Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations. IEEE Transactions on Evolutionary Computation (2018).
[22]
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1--9.
[23]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.
[24]
Xingwen Zhang, Jeff Clune, and Kenneth O Stanley. 2017. On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descent. arXiv preprint arXiv:1712.06564 (2017).
[25]
Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).

Cited By

View all
  • (2023)Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications, and Open IssuesACM Computing Surveys10.1145/360370456:2(1-34)Online publication date: 15-Sep-2023
  • (2023)A Survey on Evolutionary Neural Architecture SearchIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.310055434:2(550-570)Online publication date: Feb-2023
  • (2022)A Review on Convolutional Neural Network Encodings for NeuroevolutionIEEE Transactions on Evolutionary Computation10.1109/TEVC.2021.308863126:1(12-27)Online publication date: Feb-2022
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences
APPIS 2020: Proceedings of the 3rd International Conference on Applications of Intelligent Systems
January 2020
214 pages
ISBN:9781450376303
DOI:10.1145/3378184
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 February 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Convolutional Nets
  2. Deep learning
  3. Neuroevolution

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

APPIS 2020

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 21 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2023)Survey on Evolutionary Deep Learning: Principles, Algorithms, Applications, and Open IssuesACM Computing Surveys10.1145/360370456:2(1-34)Online publication date: 15-Sep-2023
  • (2023)A Survey on Evolutionary Neural Architecture SearchIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.310055434:2(550-570)Online publication date: Feb-2023
  • (2022)A Review on Convolutional Neural Network Encodings for NeuroevolutionIEEE Transactions on Evolutionary Computation10.1109/TEVC.2021.308863126:1(12-27)Online publication date: Feb-2022
  • (2021)Lights and shadows in Evolutionary Deep Learning: Taxonomy, critical methodological analysis, cases of study, learned lessons, recommendations and challengesInformation Fusion10.1016/j.inffus.2020.10.01467(161-194)Online publication date: Mar-2021
  • (2020)Mutational puissance assisted neuroevolutionProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3398149(1841-1848)Online publication date: 8-Jul-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media