[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content

Advertisement

Springer Nature Link
Log in

An improved weight-constrained neural network training algorithm

  • Emerging Trends of Applied Neural Computation - E_TRAINCO
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this work, we propose an improved weight-constrained neural network training algorithm, named iWCNN. The proposed algorithm exploits the numerical efficiency of the L-BFGS matrices together with a gradient-projection strategy for handling the bounds on the weights. Additionally, an attractive property of iWCNN is that it utilizes a new scaling factor for defining the initial Hessian approximation used in the L-BFGS formula. Since the L-BFGS Hessian approximation is defined utilizing a small number of correction vector pairs, our motivation is to further exploit them in order to increase the efficiency of the training algorithm and the convergence rate of the minimization process. The preliminary numerical experiments provide empirical evidence that the proposed training algorithm accelerates the training process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Al-Baali M (1998) Numerical experience with a class of self-scaling quasi-Newton algorithms. J Optim Theory Appl 96(3):533–553

    Article  MathSciNet  MATH  Google Scholar 

  2. Awan SM, Aslam M, Khan ZA, Saeed H (2014) An efficient model based on artificial bee colony optimization algorithm with neural networks for electric load forecasting. Neural Comput Appl 25(7–8):1967–1978

    Article  Google Scholar 

  3. Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148

    Article  MathSciNet  MATH  Google Scholar 

  4. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  MATH  Google Scholar 

  5. Chen W, Wang Z, Zhou J (2014) Large-scale L-BFGS using MapReduce. In: Advances in neural information processing systems, pp 1332–1340

  6. Cui K, Qin X (2018) Virtual reality research of the dynamic characteristics of soft soil under metro vibration loads based on BP neural networks. Neural Comput Appl 29(5):1233–1242

    Article  Google Scholar 

  7. Demertzis K, Iliadis L (2015) Intelligent bio-inspired detection of food borne pathogen by DNA barcodes: the case of invasive fish species Lagocephalus sceleratus. In: International conference on engineering applications of neural networks. Springer, pp 89–99

  8. Dolan E, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213

    Article  MathSciNet  MATH  Google Scholar 

  9. Dua D, Taniskidou EK (2017) UCI machine learning repository

  10. Erzin Y, Gul TO (2014) The use of neural networks for the prediction of the settlement of one-way footings on cohesionless soils based on standard penetration test. Neural Comput Appl 24(3–4):891–900

    Article  Google Scholar 

  11. Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2414–2423

  12. Horton P, Nakai K (1997) Better prediction of protein cellular localization sites with the \(k\)-nearest neighbors classifier. In: Intelligent systems in molecular biology, pp 368–383

  13. Iliadis L, Mansfield SD, Avramidis S, El-Kassaby YA (2013) Predicting Douglas-fir wood density by artificial neural networks (ANN) based on progeny testing information. Holzforschung 67(7):771–777

    Article  Google Scholar 

  14. Iliadis L, Margaritis K, Maglogiannis I (2017) Timely advances in evolving neural-based systems special issue. Evol Syst 8(1):1–2

    Article  Google Scholar 

  15. Jia F, Lei Y, Guo L, Lin J, Xing S (2018) A neural network constructed by deep learning technique and its application to intelligent fault diagnosis of machines. Neurocomputing 272:619–628

    Article  Google Scholar 

  16. Kayaer K, Yıldırım T (2003) Medical diagnosis on pima Indian diabetes using general regression neural networks. In: Proceedings of the international conference on artificial neural networks and neural information processing, pp 181–184

  17. Kostić S, Vasović D (2015) Prediction model for compressive strength of basic concrete mixture using artificial neural networks. Neural Comput Appl 26(5):1005–1024

    Article  Google Scholar 

  18. Li F, Zhang X, Zhang X, Du C, Xu Y, Tian YC (2018) Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets. Inf Sci 422:242–256

    Article  Google Scholar 

  19. Liang P, Labedan B, Riley M (2002) Physiological genomics of Escherichia coli protein families. Physiol Genom 9:15–26

    Article  Google Scholar 

  20. Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45(1–3):503–528

    Article  MathSciNet  MATH  Google Scholar 

  21. Livieris IE (2018) Improving the classification efficiency of an ANN utilizing a new training methodology. Informatics 6(1):1–17

    Article  Google Scholar 

  22. Livieris IE (2019) Forecasting economy-related data utilizing constrained recurrent neural networks. Algorithms 12:85

    Article  MathSciNet  MATH  Google Scholar 

  23. Livieris IE, Pintelas P (2012) An improved spectral conjugate gradient neural network training algorithm. Int J Artif Intell Tools 21(1):1250009

    Article  Google Scholar 

  24. Maren AJ, Harston CT, Pap RM (2014) Handbook of neural computing applications. Academic Press, Cambridge

    MATH  Google Scholar 

  25. Morales JL, Nocedal J (2011) Remark on “Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization”. ACM Trans Math Softw (TOMS) 38(1):7

    Article  MATH  Google Scholar 

  26. Moré JJ, Thuente DJ (1994) Line search algorithms with guaranteed sufficient decrease. ACM Trans Math Softw (TOMS) 20(3):286–307

    Article  MathSciNet  MATH  Google Scholar 

  27. Nguyen D, Widrow B (1990) Improving the learning speed of 2-layer neural network by choosing initial values of adaptive weights. Biol Cybern 59:71–113

    Google Scholar 

  28. Nocedal J, Wright S (2006) Numerical optimization. Springer, Berlin

    MATH  Google Scholar 

  29. Oren SS, Luenberger DG (1974) Self-scaling variable metric (ssvm) algorithms: part I: criteria and sufficient conditions for scaling a class of algorithms. Manag Sci 20(5):845–862

    Article  MATH  Google Scholar 

  30. Shanno DF, Phua KH (1978) Matrix conditioning and nonlinear optimization. Math Program 14(1):149–160

    Article  MathSciNet  MATH  Google Scholar 

  31. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  32. Yu J, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71:1054–1060

    Article  Google Scholar 

  33. Zhou B, Gao L, Dai YH (2006) Gradient methods with adaptive step-sizes. Comput Optim Appl 35(1):69–86

    Article  MathSciNet  MATH  Google Scholar 

  34. Zhu C, Byrd RH, Lu P, Nocedal J (1997) Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans Math Softw (TOMS) 23(4):550–560

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Patras, 265-00, Patras, Greece

    Ioannis E. Livieris & Panagiotis Pintelas

Authors
  1. Ioannis E. Livieris
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Panagiotis Pintelas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis E. Livieris.

Ethics declarations

Conflict of interest

The authors declared no potential conflict of interest with respect to the research, authorship and/or publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Livieris, I.E., Pintelas, P. An improved weight-constrained neural network training algorithm. Neural Comput & Applic 32, 4177–4185 (2020). https://doi.org/10.1007/s00521-019-04342-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04342-2

Keywords

Search

Navigation