[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
article

Evolving Multilayer Perceptrons

Published: 01 October 2000 Publication History

Abstract

This paper proposes a new version of a method (G-Prop, genetic backpropagation) that attempts to solve the problem of finding appropriate initial weights and learning parameters for a single hidden layer Multilayer Perceptron (MLP) by combining an evolutionary algorithm (EA) and backpropagation (BP). The EA selects the MLP initial weights, the learning rate and changes the number of neurons in the hidden layer through the application of specific genetic operators, one of which is BP training. The EA works on the initial weights and structure of the MLP, which is then trained using QuickProp; thus G-Prop combines the advantages of the global search performed by the EA over the MLP parameter space and the local search of the BP algorithm. The application of the G-Prop algorithm to several real-world and benchmark problems shows that MLPs evolved using G-Prop are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms. It also shows some improvement over previous versions of the algorithm.

References

[1]
1. Fahlman, S. E.: Faster-learning variations on back-propagation: An empirical study. Proc. 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.
[2]
2. Riedmiller, M. and Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Ruspini, H., (ed.), Proc. ICNN93, San Francisco, 1993, pp. 586-591.
[3]
3. Merelo, J. J., Patón, M., Cañas, A., Prieto A. and Morán, F.: Optimization of a competitive learning neural network by genetic algorithms. IWANN93. Lecture Notes in Computer Science, 686 (1993), 185-192.
[4]
4. Castillo, P. A., González, J., Merelo, J. J., Rivas, V., Romero, G., Prieto, A.: SA-Prop: Optimization of multilayer perceptron parameters using simulated annealing. In: Lecture Notes in Computer Science, Volume I, volume 1606, 1998, pp. 661-670.
[5]
5. Yao, X. and Liu, Y.: Towards designing artificial neural networks by evolution. Appl. Math. Comput., 91(1) (1998), 83-90.
[6]
6. Liu, Y. and Yao, X.: Evolving modular neural networks which generalise well. Proc. IEEE Int. Confer. Artificial Life and Robotics. AROBIII 98, M. Sugisake, (ed.), Vol 2, 1998, pp. 736-739.
[7]
7. Miller, G. F., Todd, P. M and Hegde, S. U.: Designing neural networks using genetic algorithms. In: J. D. Schaffer, ed., Proc. Third Int. Conf. Genetic Algorithms, San Mateo, 1989, 1989, pp 379-384.
[8]
8. Alpaydim, E.: GAL: Networks that grow when they learn and shrink when they forget. Int. J. Pattern Recognition and Artif. Intel. 8(1) (1994), 391-414.
[9]
9. Balakrishnan, K. and Honavar, V.: Evolutionary design of neural architectures - a preliminary taxonomy and guide to literature. Technical report, AI Research Group, January 1995. CS-TR 95-01.
[10]
10. Fahlman, S. and Lebière, C.: The Cascade-Correlation learning architecture. In: D. S. Touretzky, (ed.), Neural Information Processing Systems 2, Morgan-Kauffman, 1990, pp. 524-532.
[11]
11. Parekh, R., Yang, J. and Honavar, V.: Constructive neural network learning algorithms for multi-category real-valued pattern classification. Technical Report ISU-CS-TR-97-06, Department of Computer Science, Iowa State University, 1997.
[12]
12. Rathbun, T., Rogers, S., DeSimio, M. and Oxley, M.: MLP iterative construction algorithm. Neurocomputing 17 (1997), 195-216.
[13]
13. Hwang, J., You, S., Lay, S. and Jou, I.: The cascade-correlation learning: A projection pursuit learning perspective. IEEE Trans. Neural Networks 7(2) (1996), 278-289.
[14]
14. Jasic, T. and Poh, H.: Analysis of pruning in backpropagation networks for artificial and real world mapping problems. IWANN95. Lecture Notes in Computer Science 930 (1995), 239-245.
[15]
15. Pelillo, M. and Fanelli, A.: A method of pruning layered feed-forward neural networks. IWANN93. Lecture Notes in Computer Science 686 (1993), 278-283.
[16]
16. Le Cun, Y., Denker, J. S. and Solla, S. A.: Optimal brain damage. In: D. S. Touretzky, ed., Neural Information Systems 2, Morgan-Kauffman, 1990, pp. 598-605.
[17]
17. Hassibi, B., Stork, D. G., Wolff, G. and Watanabe, T.: Optimal brain surgeon: extensions and performance comparisons. In: NIPS6, 1994, pp. 263-270.
[18]
18. Yao, X.: A review of evolutionary artificial neural networks. Technical report, CSIRO, 1992.
[19]
19. Marín, F. J. and Sandoval, F.: Diseño de redes neuronales artificiales mediante algoritmos genéticos. Computación Neuronal. Universidad de Santiago de Compostela, 1995, pp. 385-424.
[20]
20. Topchy, A. P., Lebedko, O. A. and Miagkikh, V. V.: Fast learning in multilayered neural networks by means of hybrid evolutionary and gradient algorithms. To appear in Proc. of IC on Evolutionary Computation and Its Applications, Moscow, 1996.
[21]
21. De Falco, I., Iazzetta, A., Natale, P. and Tarantino, E.: Evolutionary neural networks for nonlinear dynamics modeling. Parallel Problem Solving from Nature 98. Lecture Notes in Computer Science, 1498 (1998), 593-602.
[22]
22. Whitley, D., Starkweather, T. and Bogart, C.: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Comp. 14 (1993), 347-361.
[23]
23. Schraudolph, N. N. and Belew, R. K.: Dynamic parameter encoding for genetic algorithms. Machine Learning 9(1) (1992), 9-21.
[24]
24. Castillo, P. A., González, J., Merelo, J. J., Rivas, V., Romero, G. and Prieto, A.: G-Prop: global optimization of multilayer perceptrons using GAs. Submitted to Neurocomputing, 1999.
[25]
25. Keesing, R. and Stork, D. G.: Evolution and learning in neural networks: the number and distribution of learning trials affect the rate of evolution. Adv. Neural Inform. Process. Syst. 3 (1991), 805-810.
[26]
26. Baldwin, J. M.: A new factor in evolution. Am. Naturalist 30 (1896), 441-451.
[27]
27. White, D. and Ligomenides, P.: GANNet: A genetic algorithm for optimizing topology and weights in neural network design. IWANN93. Lecture Notes in Computer Science 686 (1993), 322-327.
[28]
28. De Falco, I., Della Cioppa, A., Iazzetta, A., Natale, P. and Tarantino, E.: Optimizing neural networks for time series prediction. Third World Conference on Soft Computing (WSC3), June 1998.
[29]
29. Bebis, G., Georgiopoulos, M. and Kasparis, T.: Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization. Neurocomputing 17 (1997), 167-194.
[30]
30. Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Systems 4 (1990), 461-476.
[31]
31. Harp, S. A., Smad, T. and Guha, A.: Towards the genetic synthesis of neural networks. In: J. D. Schaffer, (ed.), Third Internat. Conf. Genetic Algorithms, Morgan Kauffmann, 1989, 360-369.
[32]
32. Dodd, N., Macfarlane, D. and Marland, C.: Optimisation of artificial neural networks structure using genetic techniques implemented on multiple transputers. Proc. Transputing 91, 1991.
[33]
33. Gruau, F. C.: Cellular enconding of genetic neural networks. Technical Report, LIP-IMAG Ecole Normale Superieure de Lyon, 46 Allée d'Italie 69007 Lyon, France, 1992.
[34]
34. Petridis, V., Kazarlis, S., Papaikonomu, A. and Filelis, A.: A hybrid genetic algorithm for training neural networks. Artificial Neural Networks 2 (1992), 953-956.
[35]
35. Castillo, P. A., González, J., Merelo, J. J., Rivas, V., Romero, G. and Prieto, A.: G-Prop-II: Global optimization of multilayer perceptrons using GAs. In: Congress on Evolutionary Computation, Washington D.C., 1999.
[36]
36. Whitley, D.: The GENITOR algorithm and selection pressure: Why rank-based allocation of reproductive trials is best. In: J. D. Schaffer, (ed.), Proc. The Third Internat. Conf. Genetic Algorithms, Morgan Kauffmann Publishers, 1989, pp. 116-121.
[37]
37. Michalewicz, Z.: Genetic Algorithms + Data Structures = Evolution Programs, Third Extended Edition. Springer-Verlag, 1996.
[38]
38. Kinnebrock, W.: Accelerating the standard backpropagation method using a genetic approach. Neurocomputing 6 (1994), 583-588.
[39]
39. Bellido, I. and Fernandez, G.: Backpropagation growing networks: Towards local minima elimination. IWANN91. Lecture Notes in Computer Science 540 (1991), 130-135.
[40]
40. Montana, D. J. and Davis. L.: Training feedforward neural networks using genetic algorithms. Proc. 11th Internat. Joint Conf. on Artif. Intel., 1989, pp. 762-767.
[41]
41. Fahlman, S.: An empirical study of learning speed in back-propagation networks. Technical report, Carnegie Mellon University, 1988.
[42]
42. Prechelt, L.: PROBEN1 -- A set of benchmarks and benchmarking rules for neural network training algorithms. Technical Report 21/94, Fakultät für Informatik, Universität Karlsruhe, Germany, September 1994.
[43]
43. Riedmiller, M.: Description and implementation details. Technical report, University of Karlsruhe, 1994.
[44]
44. Grönroos, M. A.: Evolutionary design of neural networks. MS Thesis in Computer Science, Department of Mathematical Sciences, University of Turku, 1998.
[45]
45. Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Technical report, Center for Machine Translation, Carnegie Mellon University, Pittsburgh and NEC Corporation, Tokyo, Pittsburgh, PA., 1990.
[46]
46. Mangasarian, O. L., Setiono, R. and Wolberg, W. H.: Pattern recognition via linear programming: Theory and application to medical diagnosis. In: T. F. Coleman and Y. Li, (eds), Large-scale Numerical Optimization, SIAM Publications, Philadelphia, 1990, pp. 22-30.

Cited By

View all
  • (2021)High-throughput virtual screening of small molecule inhibitors for SARS-CoV-2 protein targets with deep fusion modelsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476193(1-13)Online publication date: 14-Nov-2021
  • (2021)EvoMLP: A Framework for Evolving Multilayer PerceptronsAdvances in Computational Intelligence10.1007/978-3-030-85099-9_27(330-342)Online publication date: 16-Jun-2021
  • (2016)The Uncertainty QuandaryTransactions on Computational Collective Intelligence XXIV - Volume 977010.1007/978-3-662-53525-7_3(40-60)Online publication date: 1-Jul-2016
  • Show More Cited By

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Neural Processing Letters
Neural Processing Letters  Volume 12, Issue 2
Oct. 2000
88 pages
ISSN:1370-4621
Issue’s Table of Contents

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 October 2000

Author Tags

  1. evolutionary algorithms
  2. generalization
  3. learning
  4. neural networks
  5. optimization

Qualifiers

  • Article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2021)High-throughput virtual screening of small molecule inhibitors for SARS-CoV-2 protein targets with deep fusion modelsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476193(1-13)Online publication date: 14-Nov-2021
  • (2021)EvoMLP: A Framework for Evolving Multilayer PerceptronsAdvances in Computational Intelligence10.1007/978-3-030-85099-9_27(330-342)Online publication date: 16-Jun-2021
  • (2016)The Uncertainty QuandaryTransactions on Computational Collective Intelligence XXIV - Volume 977010.1007/978-3-662-53525-7_3(40-60)Online publication date: 1-Jul-2016
  • (2014)Adaptive water demand forecasting for near real-time management of smart water distribution systemsEnvironmental Modelling & Software10.1016/j.envsoft.2014.06.01660:C(265-276)Online publication date: 1-Oct-2014
  • (2014)Robust Neuroevolutionary Identification of Nonlinear Nonstationary ObjectsCybernetics and Systems Analysis10.1007/s10559-014-9589-550:1(17-30)Online publication date: 1-Jan-2014
  • (2011)Evolving neural networksProceedings of the Second international conference on Advances in swarm intelligence - Volume Part I10.5555/2026282.2026344(447-454)Online publication date: 12-Jun-2011
  • (2011)Evolutionary computation and its applications in neural and fuzzy systemsApplied Computational Intelligence and Soft Computing10.1155/2011/9382402011(7-7)Online publication date: 1-Jan-2011
  • (2011)Learning in the feed-forward random neural networkPerformance Evaluation10.1016/j.peva.2010.07.00668:4(361-384)Online publication date: 1-Apr-2011
  • (2011)Enhancing the classification accuracy by scatter-search-based ensemble approachApplied Soft Computing10.1016/j.asoc.2010.01.02411:1(1021-1028)Online publication date: 1-Jan-2011
  • (2009)Parameter determination and feature selection for back-propagation network by particle swarm optimizationKnowledge and Information Systems10.5555/3225629.322573321:2(249-266)Online publication date: 1-Nov-2009
  • Show More Cited By

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media