More Web Proxy on the site http://driver.im/

Article

Hyperparameter Tuning of Random Forests Using Radial Basis Function Models

Author:

Rommel G. RegisAuthors Info & Claims

Machine Learning, Optimization, and Data Science: 8th International Conference, LOD 2022, Certosa di Pontignano, Italy, September 18–22, 2022, Revised Selected Papers, Part I

Pages 309 - 324

https://doi.org/10.1007/978-3-031-25599-1_23

Published: 09 March 2023 Publication History

Abstract

This paper considers the problem of tuning the hyperparameters of a random forest (RF) algorithm, which can be formulated as a discrete black-box optimization problem. Although default settings of RF hyperparameters in software packages work well in many cases, tuning these hyperparameters can improve the predictive performance of the RF. When dealing with large data sets, the tuning of RF hyperparameters becomes a computationally expensive black-box optimization problem. A suitable approach is to use a surrogate-based method where surrogates are used to approximate the functional relationship between the hyperparameters and the overall out-of-bag (OOB) prediction error of the RF. This paper develops a surrogate-based method for discrete black-box optimization that can be used to tune RF hyperparameters. Global and local variants of the proposed method that use radial basis function (RBF) surrogates are applied to tune the RF hyperparameters for seven regression data sets that involve up to 81 predictors and up to over 21K data points. The RBF algorithms obtained better overall OOB RMSE than discrete global random search, a discrete local random search algorithm and a Bayesian optimization approach given a limited budget on the number of hyperparameter settings to consider.

References

[1]

Archetti F and Candelieri A Bayesian Optimization and Data Science 2019 Cham Springer

[2]

Bagheri S, Konen W, Emmerich M, and Bäck T Self-adjusting parameter control for surrogate-assisted constrained optimization under limited budgets Appl. Soft Comput. 2017 61 377-393

[3]

Bartz-Beielstein T and Zaefferer M Model-based methods for continuous and discrete global optimization Appl. Soft Comput. 2017 55 154-167

[4]

Bergstra J and Bengio Y Random search for hyper-parameter optimization J. Mach. Learn. Res. 2012 13 281-305

[5]

Breiman L Random forests Mach. Learn. 2001 45 5-32

[6]

Costa A and Nannicini G RBFOpt: an open-source library for black-box optimization with costly function evaluations Math. Program. Comput. 2018 10 4 597-629

[7]

De Cock, D.: Ames, Iowa: alternative to the Boston housing data as an end of semester regression project. J. Stat. Educ. 19(3) (2011)

[8]

Dua, D., Graff, C.: UCI Machine Learning Repository. School of Information and Computer Science, University of California, Irvine (2019). https://archive.ics.uci.edu/ml

[9]

Fernandes, K., Vinagre, P., Cortez, P.: A proactive intelligent decision support system for predicting the popularity of online news. In: Proceedings of the 17th EPIA 2015 - Portuguese Conference on Artificial Intelligence, Coimbra, Portugal (2015)

[10]

Hamidieh K A data-driven statistical model for predicting the critical temperature of a superconductor Comput. Mater. Sci. 2018 154 346-354

[11]

Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.: Efficient hyperparameter optimization for deep learning algorithms using deterministic RBF surrogates. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)

[12]

James G, Witten D, Hastie T, and Tibshirani R An Introduction to Statistical Learning with Applications in R 2013 New York Springer

[13]

Jones DR, Schonlau M, and Welch WJ Efficient global optimization of expensive black-box functions J. Global Optim. 1998 13 455-492

[14]

Joy, T.T., Rana, S., Gupta, S., Venkatesh, S.: Hyperparameter tuning for big data using Bayesian optimisation. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2574–2579 (2016).

[15]

Kuhn, M.: AmesHousing: the Ames Iowa housing data. R package version 0.0.4 (2020). https://CRAN.R-project.org/package=AmesHousing

[16]

Kuhn M and Johnson K Applied Predictive Modeling 2013 New York Springer

[17]

Kuhn, M., Johnson, K.: AppliedPredictiveModeling: functions and data sets for ‘applied predictive modeling’. R package version 1.1-7 (2018). https://CRAN.R-project.org/package=AppliedPredictiveModeling

[18]

Picheny, V., Ginsbourger, D., Roustant, O.: DiceOptim: Kriging-based optimization for computer experiments. R package version 2.1.1 (2021). https://CRAN.R-project.org/package=DiceOptim

[19]

Powell MJD Light W The theory of radial basis function approximation in 1990 Advances in Numerical Analysis, Volume 2: Wavelets, Subdivision Algorithms and Radial Basis Functions 1992 Oxford Oxford University Press 105-210

[20]

Probst P, Wright MN, and Boulesteix A-L Hyperparameters and tuning strategies for random forest WIREs Data Min. Knowl. Discov. 2019 9 3

[21]

R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria (2021). https://www.R-project.org/

[22]

Regis RG Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions Comput. Oper. Res. 2011 38 5 837-853

[23]

Regis, R.G.: Large-scale discrete constrained black-box optimization using radial basis functions. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2924–2931 (2020).

[24]

Roustant O, Ginsbourger D, and Deville Y DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by Kriging-based metamodeling and optimization J. Stat. Softw. 2012 51 1 1-55

[25]

RStudio Team: RStudio: Integrated Development for R. RStudio, PBC, Boston (2020). https://www.rstudio.com/

[26]

Turner, R., et al.: Bayesian optimization is superior to random search for machine learning hyperparameter tuning: analysis of the black-box optimization challenge 2020. In: Proceedings of the NeurIPS 2020 Competition and Demonstration Track, in Proceedings of Machine Learning Research, vol. 133, pp. 3–26 (2021)

[27]

Vu KK, D’Ambrosio C, Hamadi Y, and Liberti L Surrogate-based methods for black-box optimization Int. Trans. Oper. Res. 2017 24 393-424

[28]

Venables WN and Ripley BD Modern Applied Statistics with S 2002 4 New York Springer

[29]

Wright MN and Ziegler A Ranger: a fast implementation of random forests for high dimensional data in C++ and R J. Stat. Softw. 2017 77 1 1-17

[30]

Yeh IC Modeling of strength of high-performance concrete using artificial neural networks Cem. Concr. Res. 1998 28 12 1797-1808

Cited By

Heiba NMohamad YVelasco CGappa HBerlage TGeisler S(2023)Predicting and Understanding Care Levels of Elderly People with Machine LearningHCI International 2023 – Late Breaking Papers10.1007/978-3-031-48041-6_4(39-54)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-48041-6_4
Regis R(2023)Radial Basis Function and Bayesian Methods for the Hyperparameter Optimization of Classification Random ForestsComputational Science and Its Applications – ICCSA 2023 Workshops10.1007/978-3-031-37108-0_33(508-525)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-37108-0_33

Index Terms

Hyperparameter Tuning of Random Forests Using Radial Basis Function Models

Index terms have been assigned to the content through auto-classification.

Recommendations

Radial Basis Function and Bayesian Methods for the Hyperparameter Optimization of Classification Random Forests
Computational Science and Its Applications – ICCSA 2023 Workshops
Abstract
The hyperparameter optimization of a random forest (RF) is a discrete black-box optimization problem that aims to find the settings of the hyperparameters that optimize an overall out-of-bag (OOB) performance measure of the RF. This problem is ...
Global optimization via inverse distance weighting and radial basis functions
Abstract
Global optimization problems whose objective function is expensive to evaluate can be solved effectively by recursively fitting a surrogate function to function samples and minimizing an acquisition function to generate new samples. The ...
A Radial Basis Function Method for Global Optimization

We introduce a method that aims to find the global minimum of a continuous nonconvex function on a compact subset of \dR ^d. It is assumed that function evaluations are expensive and that no additional information is available. Radial basis function ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Machine Learning, Optimization, and Data Science: 8th International Conference, LOD 2022, Certosa di Pontignano, Italy, September 18–22, 2022, Revised Selected Papers, Part I

Sep 2022

638 pages

ISBN:978-3-031-25598-4

DOI:10.1007/978-3-031-25599-1

Editors:
Giuseppe Nicosia
University of Catania, Catania, Italy
,
Varun Ojha
University of Reading, Reading, UK
,
Emanuele La Malfa
University of Oxford, Oxford, UK
,
Gabriele La Malfa
University of Cambridge, Cambridge, UK
,
Panos Pardalos
University of Florida, Gainesville, FL, USA
,
Giuseppe Di Fatta
Free University of Bozen-Bolzano, Bolzano, Italy
,
Giovanni Giuffrida
University of Catania, Catania, Italy
,
Renato Umeton
Dana-Farber Cancer Institute, Boston, MA, USA

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 09 March 2023

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Heiba NMohamad YVelasco CGappa HBerlage TGeisler S(2023)Predicting and Understanding Care Levels of Elderly People with Machine LearningHCI International 2023 – Late Breaking Papers10.1007/978-3-031-48041-6_4(39-54)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-48041-6_4
Regis R(2023)Radial Basis Function and Bayesian Methods for the Hyperparameter Optimization of Classification Random ForestsComputational Science and Its Applications – ICCSA 2023 Workshops10.1007/978-3-031-37108-0_33(508-525)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-37108-0_33

View Options

View options

Media

Figures

Other

Tables

View Table of Contents