Abstract
The traditional Gradient Descent Back-propagation Neural Network Algorithm is widely used in solving many practical applications around the globe. Despite providing successful solutions, it possesses a problem of slow convergence and sometimes getting stuck at local minima. Several modifications are suggested to improve the convergence rate of Gradient Descent Backpropagation algorithm such as careful selection of initial weights and biases, learning rate, momentum, network topology, activation function and ‘gain’ value in the activation function. In a certain variation, the previous researchers demonstrated that in “feed-forward algorithm”, the slope of activation function is directly influenced by ‘gain’ parameter. This research proposed an algorithm for improving the current working performance of Back-propagation algorithm by adaptively changing the momentum value and at the same time keeping the ‘gain’ parameter fixed for all nodes in the neural network. The performance of the proposed method known as ‘Gradient Descent Method with Adaptive Momentum (GDAM)’ is compared with the performances of ‘Gradient Descent Method with Adaptive Gain (GDM-AG)’ and ‘Gradient Descent with Simple Momentum (GDM)’. The learning rate is kept fixed while sigmoid activation function is used throughout the experiments. The efficiency of the proposed method is demonstrated by simulations on three classification problems. Results show that GDAM is far better than previous methods with an accuracy ratio of 1.0 for classification problems and can be used as an alternative approach of BPNN.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Kosko, B.: Neural Network and Fuzzy Systems, 1st edn. Prentice Hall of India, Englewood Cliffs (1994)
Krasnopolsky, V.M., Chevallier, F.: Some Neural Network application in environ-mental sciences. Part II: Advancing Computational Efficiency of environmental numerical models. Neural Networks 16(3-4), 335–348 (2003)
Coppin, B.: Artificial Intelligence Illuminated, USA. Jones and Bartlet Illuminated Series, ch.11, pp. 291–324 (2004)
Basheer, I.A., Hajmeer, M.: Artificial neural networks: fundamentals, computing, design, and application. J. of Microbiological Methods 43(1), 3–31 (2000)
Zheng, H., Meng, W., Gong, B.: Neural Network and its Application on Machine fault Diagnosis. In: ICSYSE 1992, September 17-19, pp. 576–579 (1992)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Internal Representations by error Propagation. J. Parallel Distributed Processing: Explorations in the Microstructure of Cognition (1986)
Lee, K., Booth, D., Alam, P.A.: Comparison of Supervised and Unsupervised Neural Networks in Predicting Bankruptcy of Korean Firms. J. Expert Systems with Applications 16, 1–16 (2005)
Zweiri, Y.H., Seneviratne, L.D., Althoefer, K.: Stability Analysis of a Three-term Back-propagation Algorithm. J. Neural Networks 18, 1341–1347 (2005)
Fkirin, M.A., Badwai, S.M., Mohamed, S.A.: Change Detection Using Neural Network in Toshka Area. In: NSRC, 2009, Cairo, Egypt, pp. 1–10 (2009)
Sun, Y.J., Zhang, S., Miao, C.X., Li, J.M.: Improved BP Neural Network for Trans-former Fault Diagnosis. J. China University of Mining Technology. 17, 138–142 (2007)
Hamreeza, N., Nawi, N.M., Ghazali, R.: The effect of Adaptive Gain and adaptive Momentum in improving Training Time of Gradient Descent Back Propagation Algorithm on Classification problems. In: 2nd International Conference on Science Engineering and Technology, pp. 178–184 (2011)
Shao, H., Zheng, H.: A New BP Algorithm with Adaptive Momentum for FNNs Training. In: GCIS 2009, Xiamen, China, pp. 16–20 (2009)
Rehman, M.Z., Nawi, N.M., Ghazali, M.I.: Noise-Induced Hearing Loss (NIHL) Prediction in Humans Using a Modified Back Propagation Neural Network. In: 2nd International Conference on Science Engineering and Technology, pp. 185–189 (2011)
Swanston, D.J., Bishop, J.M., Mitchell, R.J.: Simple adaptive momentum: New algorithm for training multilayer Perceptrons. J. Electronic Letters 30, 1498–1500 (1994)
Mitchell, R.J.: On Simple Adaptive Momentum. In: CIS 2008, London, United Kingdom, pp. 1–6 (2008)
Nawi, N.M., Ransing, M.R., Ransing, R.S.: An improved Conjugate Gradient based learning algorithm for back propagation neural networks. J. Computational Intelligence. 4, 46–55 (2007)
Nawi, N. M.: Computational Issues in Process Optimization using historical data: PhD Eng. Thesis.Swansea University, United Kingdom (2007)
Wolberg, W.H., Mangasarian, O.L.: Multisurface method of pattern separation for medical diagnosis applied to breast cytology. National Academy of Sciences 87, 9193–9196 (1990)
Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annual Eugenics 7, 179–188 (1936)
Quinlan, J.R.: Simplifying Decision Trees. J. Man-Machine Studies 27, 221–234 (1987)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rehman, M.Z., Nawi, N.M. (2011). The Effect of Adaptive Momentum in Improving the Accuracy of Gradient Descent Back Propagation Algorithm on Classification Problems. In: Mohamad Zain, J., Wan Mohd, W.M.b., El-Qawasmeh, E. (eds) Software Engineering and Computer Systems. ICSECS 2011. Communications in Computer and Information Science, vol 179. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22170-5_33
Download citation
DOI: https://doi.org/10.1007/978-3-642-22170-5_33
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22169-9
Online ISBN: 978-3-642-22170-5
eBook Packages: Computer ScienceComputer Science (R0)