Differential Leave-One-Out Cross-Validation for Feature Selection in Generalized Linear Dependence Models
Authors: 
Alexey Morozov, Brian Angulo, Vadim Mottl, Alexander Tatarchuk, Olga KrasotkinaAuthors Info & Claims
Pages 47 - 56
Abstract
Estimation of dependencies from empirical data in a growing class of models is inevitably concerned with choosing the value of a structural parameter responsible for the model’s complexity. The most popular cross-validation schemes, in particular, LOO, suffer from the necessity to multiply repeat the model estimation on different subsamples of the training set. In this paper, we propose the method of differential LOOCV for generalized linear models of arbitrary dependencies, which allows for estimation of the model only once with each tentative value of the structural parameter. The idea of the method is that, instead of complete deleting an object from the training set at a single step of the training process, we delete only an infinitesimally small part of each of them. The indicator of the model quality is computed as the average of partial derivatives of the errors at each of single objects by the weights of their occurrence in the training set. The computing of the model quality indicator does not increase the computational complexity of the estimation procedure.
Supplementary Material
Supplemental file
- Download
- 547.91 KB
References
[1]
H Akaikei. 1973. Information theory and an extension of maximum likelihood principle. In Proc. 2nd Int. Symp. on Information Theory. 267–281.
[2]
Christopher M Bishop. 2006. Pattern recognition and machine learning. springer.
[3]
Gavin C Cawley and Nicola LC Talbot. 2008. Efficient approximate leave-one-out cross-validation for kernel logistic regression. Machine Learning 71, 2-3 (2008), 243–264.
[4]
Elena Chernousova, Nikolay Razin, Olga Krasotkina, Vadim Mottl, and David Windridge. 2014. Linear regression via Elastic Net: Non-enumerative leave-one-out verification of feature selection. In Clusters, Orders, and Trees: Methods and Applications. Springer, 377–390.
[5]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273–297.
[6]
Jerome Friedman, Trevor Hastie, and Rob Tibshirani. 2010. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software 33, 1 (2010), 1.
[7]
Michael A Golberg and Hokwon A Cho. 2004. Introduction to regression analysis. WIT press.
[8]
Todd R Golub, Donna K Slonim, Pablo Tamayo, Christine Huard, Michelle Gaasenbeek, Jill P Mesirov, Hilary Coller, Mignon L Loh, James R Downing, Mark A Caligiuri, 1999. Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. science 286, 5439 (1999), 531–537.
[9]
Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 217–226.
[10]
David JC MacKay. 1996. Hyperparameters: optimize, or integrate out?In Maximum entropy and bayesian methods. Springer, 43–59.
[11]
Rosa J Meijer and Jelle J Goeman. 2013. Efficient approximate k-fold and leave-one-out cross-validation for ridge regression. Biometrical Journal 55, 2 (2013), 141–155.
[12]
Vadim Mottl, Valentina Sulimova, Olga Krasotkina, Alexey Morozov, Alexander Tatarchuk, and Ilya Pugach. 2019. Computational Complexity of Dependence Estimation via Generalized Linear Models in Multidimensional Feature Spaces. In 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON). IEEE, 0719–0724.
[13]
John Ashworth Nelder and Robert WM Wedderburn. 1972. Generalized linear models. Journal of the Royal Statistical Society: Series A (General) 135, 3(1972), 370–384.
[14]
Mee Young Park and Trevor Hastie. 2007. L1-regularization path algorithm for generalized linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69, 4 (2007), 659–677.
[15]
Gideon Schwarz 1978. Estimating the dimension of a model. Annals of statistics 6, 2 (1978), 461–464.
[16]
William F Sharpe. 1988. Determining a fund’s effective asset mix. Investment management review 2, 6 (1988), 59–69.
[17]
William F Sharpe. 1992. Asset allocation: Management style and performance measurement. Journal of portfolio Management 18, 2 (1992), 7–19.
[18]
David J Spiegelhalter, Nicola G Best, Bradley P Carlin, and Angelika Van Der Linde. 2002. Bayesian measures of model complexity and fit. Journal of the royal statistical society: Series b (statistical methodology) 64, 4 (2002), 583–639.
[19]
William Stephenson and Tamara Broderick. 2020. Approximate cross-validation in high dimensions with guarantees. In International Conference on Artificial Intelligence and Statistics. PMLR, 2424–2434.
[20]
Alexander Tatarchuk, Vadim Mottl, Andrey Eliseyev, and David Windridge. 2008. Selectivity supervision in combining pattern-recognition modalities by feature-and kernel-selective Support Vector Machines. In 2008 19th International Conference on Pattern Recognition. IEEE, 1–4.
[21]
Brian Trippe, Jonathan Huggins, Raj Agrawal, and Tamara Broderick. 2019. LR-GLM: High-dimensional Bayesian inference using low-rank data approximations. In International Conference on Machine Learning. PMLR, 6315–6324.
[22]
VN Vapnik. 1982. Estimation of dependencies based on empirical data Springer. Information and Control(1982).
[23]
Shuaiwen Wang, Wenda Zhou, Haihao Lu, Arian Maleki, and Vahab Mirrokni. 2018. Approximate leave-one-out for fast parameter tuning in high dimensions. In International Conference on Machine Learning. PMLR, 5228–5237.
Recommendations
Feature selection via dependence maximization
We introduce a framework for feature selection based on dependence maximization between the selected features and the labels of an estimation problem, using the Hilbert-Schmidt Independence Criterion. The key idea is that good features should be highly ...
Feature selection via dependence maximization
We introduce a framework for feature selection based on dependence maximization between the selected features and the labels of an estimation problem, using the Hilbert-Schmidt Independence Criterion. The key idea is that good features should be highly ...
Feature Selection with a Linear Dependence Measure
Features which are linearly dependent on other features do not contribute toward pattern classification by linear techniques. In order to detect the linearly dependent features, a measure of linear dependence is proposed. This measure is used as an aid ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2021
126 pages
ISBN:9781450389884
DOI:10.1145/3473465
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 October 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
ITCC 2021
ITCC 2021: 2021 3rd International Conference on Information Technology and Computer Communications
June 23 - 25, 2021
Guangzhou, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 45Total Downloads
- Downloads (Last 12 months)8
- Downloads (Last 6 weeks)1
Reflects downloads up to 17 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format