More Web Proxy on the site http://driver.im/

research-article

Relational Neurogenesis for Lifelong Learning Agents

Authors:

Dhireesha KudithipudiAuthors Info & Claims

NICE '20: Proceedings of the 2020 Annual Neuro-Inspired Computational Elements Workshop

Article No.: 10, Pages 1 - 9

https://doi.org/10.1145/3381755.3381766

Published: 18 June 2020 Publication History

Abstract

Reinforcement learning systems have shown tremendous potential in being able to model meritorious behavior in virtual agents and robots. The ability to learn through continuous reinforcement and interaction with an environment negates the requirement of painstakingly curated datasets and hand crafted features. However, the ability to learn multiple tasks in a sequential manner, referred to as lifelong or continual learning, remains unresolved. Current implementations either concentrate on preserving information in fixed capacity networks, or propose incrementally growing networks which randomly search through an unconstrained solution space. This work proposes a novel algorithm for continual learning using neurogenesis in reinforcement learning agents. It builds upon existing neuroevolutionary techniques, and incorporates several new mechanisms for limiting the memory resources while expanding neural network learning capacity. The algorithm is tested on a custom set of sequential virtual environments which emulate meaningful and relevant scenarios.

References

[1]

Ratan D Bhardwaj, Maurice A Curtis, Kirsty L Spalding, Bruce A Buchholz, David Fink, Thomas Björk-Eriksson, Claes Nordborg, Fred H Gage, Henrik Druid, Peter S Eriksson, et al. 2006. Neocortical neurogenesis in humans is restricted to development. Proceedings of the National Academy of Sciences 103, 33 (2006), 12564--12568.

[2]

George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems 2, 4 (1989), 303--314.

[3]

Timothy J Draelos, Nadine E Miner, Christopher C Lamb, Jonathan A Cox, Craig M Vineyard, Kristofor D Carlson, William M Severa, Conrad D James, and James B Aimone. 2017. Neurogenesis deep learning: Extending deep networks to accommodate new classes. In 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, 526--533.

[4]

Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. 2017. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 (2017).

[5]

Michael E Hasselmo, Eric Schnell, and Edi Barkai. 1995. Dynamics of learning and recall at excitatory recurrent synapses and cholinergic modulation in rat hippocampal region CA3. Journal of Neuroscience 15, 7 (1995), 5249--5262.

[6]

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences (2017), 201611835.

[7]

Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).

[8]

Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).

[9]

David Lopez-Paz et al. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems. 6467--6476.

[10]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).

[11]

Justin K Pugh and Kenneth O Stanley. 2013. Evolving multimodal controllers with HyperNEAT. In Proceedings of the 15th annual conference on Genetic and evolutionary computation. ACM, 735--742.

Digital Library

[12]

Mark Bishop Ring. 1994. Continual learning in reinforcement environments. Ph.D. Dissertation. University of Texas at Austin Austin, Texas 78712.

[13]

Mark B Ring. 1997. CHILD: A first step towards continual learning. Machine Learning 28, 1 (1997), 77--104.

Digital Library

[14]

Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016).

[15]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[16]

Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & Compress: A scalable framework for continual learning. arXiv preprint arXiv:1805.06370 (2018).

[17]

Lisa B Soros and Kenneth O Stanley. 2014. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In H. Sayama, J. Rieffel, S. Risi, R. Doursat, & H. Lipson (Eds.), Artificial life 14: Proceedings of the Fourteenth International Conference on the Synthesis and Simulation of Living Systems. Citeseer, 793--800.

[18]

Kirsty L Spalding, Olaf Bergmann, Kanar Alkass, Samuel Bernard, Mehran Salehpour, Hagen B Huttner, Emil Boström, Isabelle Westerlund, Céline Vial, Bruce A Buchholz, et al. 2013. Dynamics of hippocampal neurogenesis in adult humans. Cell 153, 6 (2013), 1219--1227.

[19]

Kenneth O Stanley and Risto Miikkulainen. 2002. Efficient evolution of neural network topologies. In Evolutionary Computation, 2002. CEC'02. Proceedings of the 2002 Congress on, Vol. 2. IEEE, 1757--1762.

[20]

Unity Engine. [n.d.]. Unity3D. https://unity3d.com/

[21]

Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. 2018. Relational Deep Reinforcement Learning. arXiv preprint arXiv:1806.01830 (2018).

Cited By

Rohlfs C(2025)Generalization in neural networks: A broad surveyNeurocomputing10.1016/j.neucom.2024.128701611(128701)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128701
Khodaee PViktor HMichalowski W(2024)Knowledge transfer in lifelong machine learning: a systematic literature reviewArtificial Intelligence Review10.1007/s10462-024-10853-957:8Online publication date: 26-Jul-2024
https://doi.org/10.1007/s10462-024-10853-9
Daram AKudithipudi D(2023)NEO: Neuron State Dependent Mechanisms for Efficient Continual LearningProceedings of the 2023 Annual Neuro-Inspired Computational Elements Conference10.1145/3584954.3584960(11-19)Online publication date: 11-Apr-2023
https://dl.acm.org/doi/10.1145/3584954.3584960
Show More Cited By

Index Terms

Relational Neurogenesis for Lifelong Learning Agents
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
        Lifelong machine learning
    2. Machine learning approaches
      1. Bio-inspired approaches
        Genetic algorithms
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Reinforcement learning

Recommendations

Scalable lifelong reinforcement learning

Deriving a novel scalable algorithm for lifelong policy search in reinforcement learning.Acquiring linear convergence rate of our new algorithmDemonstrating the effectiveness of our technique dynamical systems and learning speed-ups on unobserved tasks. ...
Lifelong Machine Learning
Relational Reinforcement Learning

Relational reinforcement learning is presented, a learning technique that combines reinforcement learning with relational learning or inductive logic programming. Due to the use of a more expressive representation language to represent states, actions ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

NICE '20: Proceedings of the 2020 Annual Neuro-Inspired Computational Elements Workshop

March 2020

131 pages

ISBN:9781450377188

DOI:10.1145/3381755

Conference Chair:
Murat Okandan,
Program Chair:
James B. Aimone

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

INTEL: Intel Corporation
IBM: IBM

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

NICE '20

NICE '20: Neuro-inspired Computational Elements Workshop

March 17 - 20, 2020

Heidelberg, Germany

Acceptance Rates

Overall Acceptance Rate 25 of 40 submissions, 63%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
168
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)11

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rohlfs C(2025)Generalization in neural networks: A broad surveyNeurocomputing10.1016/j.neucom.2024.128701611(128701)Online publication date: Jan-2025
https://doi.org/10.1016/j.neucom.2024.128701
Khodaee PViktor HMichalowski W(2024)Knowledge transfer in lifelong machine learning: a systematic literature reviewArtificial Intelligence Review10.1007/s10462-024-10853-957:8Online publication date: 26-Jul-2024
https://doi.org/10.1007/s10462-024-10853-9
Daram AKudithipudi D(2023)NEO: Neuron State Dependent Mechanisms for Efficient Continual LearningProceedings of the 2023 Annual Neuro-Inspired Computational Elements Conference10.1145/3584954.3584960(11-19)Online publication date: 11-Apr-2023
https://dl.acm.org/doi/10.1145/3584954.3584960
Kudithipudi DDaram AZyarah AZohora FAimone JYanguas-Gil ASoures NNeftci EMattina MLomonaco VThiem CEpstein B(2023)Design principles for lifelong learning AI acceleratorsNature Electronics10.1038/s41928-023-01054-36:11(807-822)Online publication date: 16-Nov-2023
https://doi.org/10.1038/s41928-023-01054-3
Rohlfs C(2023)A descriptive analysis of olfactory sensation and memory in Drosophila and its relation to artificial neural networksNeurocomputing10.1016/j.neucom.2022.10.068518:C(15-29)Online publication date: 21-Jan-2023
https://dl.acm.org/doi/10.1016/j.neucom.2022.10.068
Kudithipudi DAguilar-Simon MBabb JBazhenov MBlackiston DBongard JBrna AChakravarthi Raja SCheney NClune JDaram AFusi SHelfer PKay LKetz NKira ZKolouri SKrichmar JKriegman SLevin MMadireddy SManicka SMarjaninejad AMcNaughton BMiikkulainen RNavratilova ZPandit TParker APilly PRisi SSejnowski TSoltoggio ASoures NTolias AUrbina-Meléndez DValero-Cuevas Fvan de Ven GVogelstein JWang FWeiss RYanguas-Gil AZou XSiegelmann H(2022)Biological underpinnings for lifelong learning machinesNature Machine Intelligence10.1038/s42256-022-00452-04:3(196-210)Online publication date: 23-Mar-2022
https://doi.org/10.1038/s42256-022-00452-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten