Abstract
In this work we propose the use of mutual information to measure the predictive potential of principal components in regression. We show that this criterion produces the same results as previous works which used the correlation to measure the strength of the relationship between the response variable with the extracted principal components in Gaussian settings. We demonstrate this in the linear regression model and also beyond that, in the conditional mean model and the conditional independence model, two common choices in sufficient dimension reduction, achieving a connection between unsupervised and supervised dimension reduction methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
A. Artemiou, B. Li, On principal components and regression: a statistical explanation of a natural phenomenon. Stat. Sin. 19, 1557–1565 (2009)
A. Artemiou, B. Li, Predictive power of principal components for single-index model and sufficient dimension reduction. J. Multivar. Anal. 119, 176–184 (2013)
R.D. Cook, Fisher lecture: dimension reduction in regression. Stat. Sci. 22, 1–40 (2007)
R.D. Cook, B. Li, Dimension Reduction for the conditional mean. Ann. Stat. 30, 455–474 (2002)
D.R. Cox, Notes on some aspects of regression analysis. J. R. Stat. Soc. Ser. A 131, 265–279 (1968)
P. Diaconis, D. Freedman, Asymptotics of graphical projection pursuit. Ann. Stat. 12, 793–815 (1984)
A.S. Hadi, R.F. Ling, Some cautionary notes on the use of principal components in regression. Am. Stat. 52, 15–19 (1998)
P. Hall, Y.J. Yang, Ordering and selecting components in multivariate or functional data linear prediction. J. R. Stat. Soc. Ser. B: Stat. Methodol. 72, 93–110 (2010)
H. Hotelling, Analysis of a complex statistical variable into its principal components. J. Educ. Psychol. 24, 417–441 (1933)
I.T. Joliffe, A note on the use of principal components in regression. Appl. Stat. 31, 300–303 (1982)
I.T. Joliffe, Principal Component Analysis, 2nd edn. (Springer, New York, 2002)
B. Jones, A. Artemiou, On principal component regression with Hilbertian predictors. Ann. Inst. Stat. Math. 72, 627–644 (2020)
B. Jones, A. Artemiou, Revisiting the predictive potential of Kernel principal components. Stat. Probab. Lett. 171 (2021). (10.1016/j.spl.2020.109019)
B. Jones, A. Artemiou, B. Li, On the predictive potential of kernel principal components. Electron. J. Stat. 14, 1–23 (2020)
B. Li, Comment: Fisher lecture: dimension reduction in regression. Stat. Sci. 22, 32–35 (2007)
B. Li, Sufficient Dimension Reduction: Methods and Applications with R, 1st edn. (CRC Press, Boca Raton, 2018)
F. Mosteller, J.W. Tukey, Data Analysis and Regression (Addison-Wesley, Reading, MA, 1977).
L. Ni, Principal component regression revisited. Stat. Sin. 21, 741–747 (2011)
L. Paninski, Estimation of entropy and mutual information. Neural Computat. 15(6), 1191–1253 (2003)
K. Pearson, On lines and planes of closest fit to a system of points in space. Philos. Mag. (6), 2, 559–572 (1901)
Acknowledgements
The author would like to thank the editors and two anonymous reviewers for their comments. I would also like to thank Prof. Bing Li for an insightful discussion a few years back. Finally, special thanks to Prof. R. D. Cook for his contributions and his kindness. Back in time, I was still a PhD student, and after publishing the first paper on this topic (Artemiou and Li (2009)—my MSc thesis paper), I met him for the first time in the summer of 2009 at the JSM conference in Washington DC. When I introduced myself, his immediate reaction was “The principal component guy.” It stuck with me since then and anytime I talk about principal components, his reaction comes to mind. It is that reaction that convinced me to contribute something on the predictive potential of PCA for this edition. I also want to thank him because, although he was surrounded by a number of his students and collaborators, he took a couple of steps away from them and a couple of minutes to discuss with me. I found it very kind of him.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Artemiou, A. (2021). Using Mutual Information to Measure the Predictive Power of Principal Components. In: Bura, E., Li, B. (eds) Festschrift in Honor of R. Dennis Cook. Springer, Cham. https://doi.org/10.1007/978-3-030-69009-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-69009-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69008-3
Online ISBN: 978-3-030-69009-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)