More Web Proxy on the site http://driver.im/

research-article

Transfer learning through indirect encoding

Authors:

Phillip Verbancsics,

Kenneth O. StanleyAuthors Info & Claims

GECCO '10: Proceedings of the 12th annual conference on Genetic and evolutionary computation

Pages 547 - 554

https://doi.org/10.1145/1830483.1830587

Published: 07 July 2010 Publication History

Abstract

An important goal for the generative and developmental systems (GDS) community is to show that GDS approaches can compete with more mainstream approaches in machine learning (ML). One popular ML domain is RoboCup and its subtasks (e.g. Keepaway). This paper shows how a GDS approach called HyperNEAT competes with the best results to date in Keepaway. Furthermore, a significant advantage of GDS is shown to be in transfer learning. For example, playing Keepaway should contribute to learning the full game of soccer. Previous approaches to transfer have focused on transforming the original representation to fit the new task. In contrast, this paper explores transfer with a representation designed to be the same even across different tasks. A bird's eye view (BEV) representation is introduced that can represent different tasks on the same two-dimensional map. Yet the problem is that a raw two-dimensional map is high-dimensional and unstructured. The problem is addressed naturally by indirect encoding, which compresses the representation in HyperNEAT by exploiting its geometry. The result is that the BEV learns a Keepaway policy that transfers from two different training domains without further learning or manipulation. The results in this paper thus show the power of GDS versus other ML methods.

References

[1]

Charu C. Aggarwal. On k-anonymity and the curse of dimensionality. In Proc. of the 31st Int. Conf. on Very Large DB, pages 901--909. VLDB Endowment, 2005.

Digital Library

[2]

R. Bellman. Adaptive Control Processes: A Guided Tour. Princeton University Press, 1961.

[3]

Rich Caruana. Multitask learning. In Machine Learning, pages 41--75, 1997.

Digital Library

[4]

Jeff Clune, Benjamin E. Beckmann, Charles Ofria, and Robert T. Pennock. Evolving coordinated quadruped gaits with the hyperneat generative encoding. In Proc. of the IEEE Congress on Ev. Comp. Special Section on Ev. Robotics, Piscataway, NJ, USA, 2009. IEEE Press.

Digital Library

[5]

David D'Ambroiso and Kenneth O. Stanley. Evolving policy geometry for scalable multiagent learning. In Proc. of the 9th Int. Conf. on AAMAS, page 8, New York, NY, USA, 2010. ACM Press.

Digital Library

[6]

David B. D'Ambrosio and Kenneth O. Stanley. Generative encoding for multiagent learning. In Proc. of the Genetic and Ev. Comp. Conf., New York, NY, 2008. ACM Press.

Digital Library

[7]

Jerome H. Friedman. On bias, variance, 0/1 - loss, and the curse-of-dimensionality. Data Min. Knowl. Discov., 1(1):55--77, 1997.

Digital Library

[8]

Jason Gauci and Kenneth O. Stanley. A case study on the critical role of geometric regularity in machine learning. In Proc. of the 23rd AAAI Conf. on AI, Menlo Park, CA, 2008. AAAI Press.

Digital Library

[9]

Jason Gauci and Kenneth O. Stanley. Autonomous evolution of topographic regularities in artificial neural networks. Neural Computation, page 38, 2010. To appear.

Digital Library

[10]

Benjamin Kuipers. The spatial semantic heirarchy. Artical Intelligence, 119:191--233, 2000.

Digital Library

[11]

Madhubanti Maitra and Amitava Chatterjee. A hybrid cooperative-comprehensive learning based pso algorithm for image segmentation using multilevel thresholding. Expert Systems with Applications, 34(2):1341 - 1350, 2008.

Digital Library

[12]

Jan FH. Metzen, Mark Edgington, Yohannes Kassahun, and Frank Kirchner. Performance evaluation of EANT in the robocup keepaway benchmark. In Proc. of the 6th Int. Conf. on ML and Apps., pages 342--347, Washington, DC, USA, 2007. IEEE Computer Society.

Digital Library

[13]

Itsuki Noda, Hitoshi Matsubara, Kazuo Hiraki, and Ian Frank. Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence, 12:233--250, 1998.

[14]

G. A. Rummery and M. Niranjan. On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG-RT 116, Cambridge Uni. Eng. Dept., 1994.

[15]

Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines Special Issue on Developmental Systems, 8(2):131--162, 2007.

Digital Library

[16]

Kenneth O. Stanley, David B. D'Ambrosio, and Jason Gauci. A hypercube-based indirect encoding for evolving large-scale neural networks. Artificial Life, 15, 2009.

Digital Library

[17]

Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10:99--127, 2002.

Digital Library

[18]

Kenneth O. Stanley and Risto Miikkulainen. A taxonomy for artificial embryogeny. Artificial Life, 9(2):93--130, 2003.

Digital Library

[19]

Kenneth O. Stanley and Risto Miikkulainen. Competitive coevolution through evolutionary complexification. Journal of Artificial Intelligence Research, 21:63--100, 2004.

Digital Library

[20]

Peter Stone, Gregory Kuhlmann, Matthew E. Taylor, and Yaxin Liu. Keepaway soccer: From machine learning testbed to benchmark. In Robot Soccer World Cup IX, pages 93--105. Springer Verlag, 2006.

Digital Library

[21]

Peter Stone and Richard S. Sutton. Scaling reinforcement learning to robocup soccer. In The 18th Int. Conf. on ML, pages 537--544, New York, NY, June 2001, ACM.

Digital Library

[22]

Peter Stone and Richard S. Sutton. Keepaway soccer: A machine learning testbed. In Robot Soccer World Cup V, pages 214--223, London, UK, 2002. Springer-Verlag.

Digital Library

[23]

Peter Stone, Richard S. Sutton, and Gregory Kuhlmann. Reinforcement learning for RoboCup-soccer keepaway. Adaptive Behavior, 13(3):165--188, 2005.

[24]

Peter Stone and Manuela Veloso. Layered learning. In Ramon López de Mántaras and Enric Plaza, editors, Proc. of the 11th European Conf. on ML, pages 369--381. Springer Verlag, Barcelona,Spain, May/June 2000.

Digital Library

[25]

Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems 8, pages 1038--1044. MIT Press, 1996.

[26]

Erik Talvitie and Satinder Singh. An experts algorithm for transfer learning. In Proc. of the 10th Int. Joint Conf. on AI, pages 1065--1070, 2007.

Digital Library

[27]

Matthew E. Taylor and Peter Stone. Cross-domain transfer for reinforcement learning. In Proc. of the 24th Int. Conf. on ML, pages 879--886, New York, NY, USA, 2007. ACM.

Digital Library

[28]

Matthew E. Taylor, Peter Stone, and Yaxin Liu. Transfer learning vis inter-task mappings for temporal difference learning. Journal of Machine Learning Research, 1(8):2125--2167, September 2007.

Digital Library

[29]

Matthew E. Taylor, Shimon Whiteson, and Peter Stone. Comparing evolutionary and temporal difference methods in a reinforcement learning domain. In Proc. of the Genetic and Ev. Comp. Conf., pages 1321--1328, New York, NY, July 2006. ACM Press.

Digital Library

[30]

Matthew E. Taylor, Shimone Whiteson, and Peter Stone. Transfer via intertask mappings in policy search reinforcement learning. In Proc. of the AAMAS Conf., New York, NY, May 2007, ACM Press.

Digital Library

[31]

A. M. Turing. The Chemical Basis of Morphogenesis. Royal Society of London Philosophical Transactions Series B, 237:37--72, August 1952.

[32]

Shimon Whiteson. Improving reinforcement learning function approximators via neuroevolution. In Proc. of the 4th Int. Joint Conf. on AAMAS, pages 1386--1386, New York, NY, USA, 2005. ACM.

Digital Library

[33]

Shimon Whiteson and Daniel Whiteson. Stochastic optimization for collision selection in high energy physics. In Proc. of the 19th Annual Innov. Apps. of AI Conf., Vancouver, Canada, July 2007. AAAI Press.

Digital Library

Cited By

Schrum JMiikkulainen R(2024)Constructing Game Agents Through Simulated EvolutionEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_15(457-466)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_15
Szemenyei MEstivill-Castro V(2021)Fully neural object detection solutions for robot soccerNeural Computing and Applications10.1007/s00521-021-05972-134:24(21419-21432)Online publication date: 15-Apr-2021
https://doi.org/10.1007/s00521-021-05972-1
Helms LClune J(2017)Improving HybrID: How to best combine indirect and direct encoding in evolutionary algorithmsPLOS ONE10.1371/journal.pone.017463512:3(e0174635)Online publication date: 23-Mar-2017
https://doi.org/10.1371/journal.pone.0174635
Show More Cited By

Index Terms

Transfer learning through indirect encoding
1. Computing methodologies
  1. Machine learning

Recommendations

Policy transfer methods in RoboCup keep-away
GECCO '18: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Policy (behavior) transfer is a method to speed-up and improve learning by leveraging knowledge from learning in related but simpler tasks. That is, learned information is reused and shared between a source and target tasks, where target tasks were used ...
Knowledge Transfer from Keepaway Soccer to Half-field Offense through Program Symbiosis: Building Simple Programs for a Complex Task
GECCO '15: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation

Half-field Offense (HFO) is a sub-task of Robocup 2D Simulated Soccer. HFO is a challenging, multi-agent machine learning problem in which a team of offense players attempt to manoeuvre the ball past a defending team and around the goalie in order to ...
An enhanced hypercube-based encoding for evolving the placement, density, and connectivity of neurons

Intelligence in nature is the product of living brains, which are themselves the product of natural evolution. Although researchers in the field of neuroevolution NE attempt to recapitulate this process, artificial neural networks ANNs so far evolved ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '10: Proceedings of the 12th annual conference on Genetic and evolutionary computation

July 2010

1520 pages

ISBN:9781450300728

DOI:10.1145/1830483

General Chair:
Martin Pelikan
University of Missouri, USA
,
Program Chair:
Jürgen Branke
University of Warwick, Coventry, UK

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '10

Sponsor:

SIGEVO

GECCO '10: Genetic and Evolutionary Computation Conference

July 7 - 11, 2010

Oregon, Portland, USA

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
187
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 23 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schrum JMiikkulainen R(2024)Constructing Game Agents Through Simulated EvolutionEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_15(457-466)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_15
Szemenyei MEstivill-Castro V(2021)Fully neural object detection solutions for robot soccerNeural Computing and Applications10.1007/s00521-021-05972-134:24(21419-21432)Online publication date: 15-Apr-2021
https://doi.org/10.1007/s00521-021-05972-1
Helms LClune J(2017)Improving HybrID: How to best combine indirect and direct encoding in evolutionary algorithmsPLOS ONE10.1371/journal.pone.017463512:3(e0174635)Online publication date: 23-Mar-2017
https://doi.org/10.1371/journal.pone.0174635
Gillespie LGonzalez GSchrum JBosman P(2017)Comparing direct and indirect encodings using both raw and hand-designed features in tetrisProceedings of the Genetic and Evolutionary Computation Conference10.1145/3071178.3071195(179-186)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.1145/3071178.3071195
Schrum JMiikkulainen R(2015)Constructing Game Agents Through Simulated EvolutionEncyclopedia of Computer Graphics and Games10.1007/978-3-319-08234-9_15-1(1-10)Online publication date: 4-Dec-2015
https://doi.org/10.1007/978-3-319-08234-9_15-1
D’Ambrosio DGauci JStanley K(2014)HyperNEAT: The First Five YearsGrowing Adaptive Machines10.1007/978-3-642-55337-0_5(159-185)Online publication date: 5-Jun-2014
https://doi.org/10.1007/978-3-642-55337-0_5
D’Ambrosio DStanley K(2013)Scalable multiagent learning through indirect encoding of policy geometryEvolutionary Intelligence10.1007/s12065-012-0086-36:1(1-26)Online publication date: 18-Jan-2013
https://doi.org/10.1007/s12065-012-0086-3
Inden BJin YHaschke RRitter H(2012)Evolving neural fields for problems with large input and output spacesNeural Networks10.1016/j.neunet.2012.01.00128(24-39)Online publication date: Apr-2012
https://doi.org/10.1016/j.neunet.2012.01.001
Clune JStanley KPennock ROfria C(2011)On the Performance of Indirect Encoding Across the Continuum of RegularityIEEE Transactions on Evolutionary Computation10.1109/TEVC.2010.210415715:3(346-367)Online publication date: 1-Jun-2011
https://dl.acm.org/doi/10.1109/TEVC.2010.2104157
Suchorzewski M(2011)Evolving scalable and modular adaptive networks with Developmental Symbolic EncodingEvolutionary Intelligence10.1007/s12065-011-0057-04:3(145-163)Online publication date: 3-May-2011
https://doi.org/10.1007/s12065-011-0057-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents