A Novel Regularization Paradigm for the Extreme Learning Machine
Authors: 
Yuao Zhang, Yunwei Dai, Qingbiao WuAuthors Info & Claims
Pages 7009 - 7033
Abstract
Due to its fast training speed and powerful approximation capabilities, the extreme learning machine (ELM) has generated a lot of attention in recent years. However, the basic ELM still has some drawbacks, such as the tendency to over-fitting and the susceptibility to noisy data. By adding a regularization term to the basic ELM, the regularized extreme learning machine (R-ELM) can dramatically improve its generalization and stability. In the R-ELM, choosing an appropriate regularization parameter is critical since it can regulate the fitting and generalization capabilities of the model. In this paper, we propose the regularized functional extreme learning machine (RF-ELM), which employs the regularization functional instead of a preset regularization parameter for adaptively choosing appropriate regularization parameters. The regularization functional is defined according to output weights, and the successive approximation iterative algorithm is utilized to solve the output weights so that we can get their values simultaneously at each iteration step. We also developed a parallel version of RF-ELM (PRF-ELM) to deal with big data tasks. Furthermore, the analysis of convexity and convergence ensures the validity of the model training. Finally, the experiments on the function approximation and the UCI repository with or without noise data demonstrate the superiority and competitiveness of our proposed models.
References
[1]
Oussous A, Benjelloun F-Z, Lahcen AA, and Belfkih S Big data technologies: a survey J King Saud Univer-Comp Inf Sci 2018 30 4 431-448
[2]
Rastogi R, Sharma S, and Chandra S Robust parametric twin support vector machine for pattern classification Neural Process Lett 2018 47 1 293-323
[3]
Khan NM and Khan GM Real-time lossy audio signal reconstruction using novel sliding based multi-instance linear regression/random forest and enhanced cgpann Neural Process Lett 2021 53 1 227-255
[4]
Pandey SK and Janghel RR Recent deep learning techniques, challenges and its applications for medical healthcare system: a review Neural Process Lett 2019 50 2 1907-1935
[5]
Lent R A generalized reinforcement learning scheme for random neural networks Neural Comp Appl 2019 31 7 2699-2716
[6]
Ke S and Liu W Consistency of multiagent distributed generative adversarial networks IEEE Trans Cybern 2020 52 6 4886-4896
[7]
Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541), vol. 2, IEEE pp. 985–990
[8]
Huang G-B, Zhu Q-Y, and Siew C-K Extreme learning machine: theory and applications Neurocomputing 2006 70 1–3 489-501
[9]
Huang G-B, Wang DH, and Lan Y Extreme learning machines: a survey Int J Mach Learn Cybern 2011 2 2 107-122
[10]
Zhu S, Wang H, Lv H, and Zhang H Augmented online sequential quaternion extreme learning machine Neural Process Lett 2021 53 2 1161-1186
[11]
Ibrahim W and Abadeh MS Protein fold recognition using deep kernelized extreme learning machine and linear discriminant analysis Neural Comp Appl 2019 31 8 4201-4214
[12]
Li L, Zhao K, Sun R, Gan J, Yuan G, and Liu T Parameter-free extreme learning machine for imbalanced classification Neural Process Lett 2020 52 3 1927-1944
[13]
Ren W and Han M Classification of EEG signals using hybrid feature extraction and ensemble extreme learning machine Neural Process Lett 2019 50 2 1281-1301
[14]
Xia J, Yang D, Zhou H, Chen Y, Zhang H, Liu T, Heidari AA, Chen H, and Pan Z Evolving kernel extreme learning machine for medical diagnosis via a disperse foraging sine cosine algorithm Comp Biol Med 2022 141 105137
[15]
El Bourakadi D, Yahyaouy A, and Boumhidi J Improved extreme learning machine with AutoEncoder and particle swarm optimization for short-term wind power prediction Neural Comput Appl 2022 34 6 4643-4659
[16]
Li Y, Zhang S, Yin Y, Zhang J, and Xiao W A soft sensing scheme of gas utilization ratio prediction for blast furnace via improved extreme learning machine Neural Process Lett 2019 50 2 1191-1213
[17]
Ma Z and Dai Q Selected an stacking elms for time series prediction Neural Process Lett 2016 44 3 831-856
[18]
Raghuwanshi BS and Shukla S Generalized class-specific kernelized extreme learning machine for multiclass imbalanced learning Expert Sys Appl 2019 121 244-255
[19]
Zou W, Yao F, Zhang B, and Guan Z Improved meta-elm with error feedback incremental elm as hidden nodes Neural Comp Appl 2018 30 11 3363-3370
[20]
Yang Y, Wang Y, and Yuan X Parallel chaos search based incremental extreme learning machine Neural Process Lett 2013 37 3 277-301
[21]
Huang G, Huang G-B, Song S, and You K Trends in extreme learning machines: a review Neural Netw 2015 61 32-48
[22]
Scardapane S and Wang D Randomness in neural networks: an overview Wiley Interdiscipl Rev: Data Min Knowl Discovery 2017 7 2 1200
[23]
Markowska-Kaczmar U and Kosturek M Extreme learning machine versus classical feedforward network: comparison from the usability perspective Neural Comput Appl 2021 33 22 15121-15144
[24]
Freire AL, Rocha-Neto AR, and Barreto GA On robust randomized neural networks for regression: a comprehensive review and evaluation Neural Comp Appl 2020 32 22 16931-16950
[25]
Ding S, Zhao H, Zhang Y, Xu X, and Nie R Extreme learning machine: algorithm, theory and applications Artif Intell Rev 2015 44 1 103-115
[26]
Zhang T, Deng Z, Choi K-S, Liu J, Wang S (2017) Robust extreme learning fuzzy systems using ridge regression for small and noisy datasets. In: 2017 IEEE International conference on fuzzy systems (FUZZ-IEEE), pp. 1–7
[27]
Yildirim H and Özkale MR The performance of elm based ridge regression via the regularization parameters Expert Sys Appl 2019 134 225-233
[28]
Kärkkäinen T Extreme minimal learning machine: Ridge regression with distance-based basis Neurocomputing 2019 342 33-48
[29]
Kang MG and Katsaggelos AK General choice of the regularization functional in regularized image restoration IEEE Trans Image Process 1995 4 5 594-602
[30]
Kang MG Generalized multichannel image deconvolution approach and its applications Opt Eng 1998 37 11 2953-2964
[31]
Haber E and Tenorio L Learning regularization functionals-a supervised training approach Inver Prob 2003 19 3 611
[32]
Zhang B, Ma Z, Liu Y, Yuan H, and Sun L Ensemble based reactivated regularization extreme learning machine for classification Neurocomputing 2018 275 255-266
[33]
Wang X-B, Zhang X, Li Z, and Wu J Ensemble extreme learning machines for compound-fault diagnosis of rotating machinery Knowl-Based Sys 2020 188 105012
[34]
Li G and Zou J Multi-parallel extreme learning machine with excitatory and inhibitory neurons for regression Neural Process Lett 2020 51 2 1579-1597
[35]
Wang Y, Dou Y, Liu X, and Lei Y Pr-elm: Parallel regularized extreme learning machine based on cluster Neurocomputing 2016 173 1073-1081
[36]
Duan M, Li K, Liao X, and Li K A parallel multiclassification algorithm for big data using an extreme learning machine IEEE Trans Neural Netw Learn Syst 2017 29 6 2337-2351
[37]
Yao L and Ge Z Distributed parallel deep learning of hierarchical extreme learning machine for multimode quality prediction with big process data Eng Appl Artif Intell 2019 81 450-465
[38]
Dokeroglu T and Sevinc E Evolutionary parallel extreme learning machines for the data classification problem Comp Ind Eng 2019 130 237-249
[39]
Rumelhart DE, Hinton GE, and Williams RJ Learning representations by back-propagating errors Nature 1986 323 533-536
[40]
Marquardt DW An algorithm for least-squares estimation of nonlinear parameters J Soci Ind Appl Math 1963 11 2 431-441
[41]
Martínez-Martínez JM, Escandell-Montero P, Soria-Olivas E, Martín-Guerrero JD, Magdalena-Benedito R, and GóMez-Sanchis J Regularized extreme learning machine for regression problems Neurocomputing 2011 74 17 3716-3721
[42]
Dua D, Graff C (2019) UCI machine learning repository. https://archive.ics.uci.edu/ml. Accessed 8 December 2021
[43]
Zhang K and Luo M Outlier-robust extreme learning machine for regression problems Neurocomputing 2015 151 1519-1527
Recommendations
Sparse and Outlier Robust Extreme Learning Machine Based on the Alternating Direction Method of Multipliers
AbstractExtreme learning machine (ELM) has been extensively researched for its fast training speed and powerful learning abilities. Entering the era of big data, large-scale learning tasks, the universality of noisy data and data with distributed storage ...
Manifold regularized extreme learning machine
Extreme learning machine (ELM) works for generalized single-hidden-layer feedforward networks (SLFNs), and its essence is that the hidden layer of SLFNs need not be tuned. But ELM only utilizes labeled data to carry out the supervised learning task. In ...
Fast sparse approximation of extreme learning machine
We introduce a fast sparse approximation schemes of extreme learning machine (ELM) named FSA-ELM of extreme learning machine (ELM). Our algorithms have two compelling features: low complexity and sparse solution. Experiments on benchmark data sets show ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 10 April 2023
Accepted: 10 March 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Jan 2025