Article

Linear discriminant analysis: new formulations and overfit analysis

Authors:

Dijun Luo,

Chris Ding,

Heng HuangAuthors Info & Claims

AAAI'11: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence

Pages 417 - 422

Published: 07 August 2011 Publication History

Publisher Site

Abstract

In this paper, we will present a unified view for LDA. We will (1) emphasize that standard LDA solutions are not unique, (2) propose several new LDA formulations: St-orthonormal LDA, Sw-orthonormal LDA and orthogonal LDA which have unique solutions, and (3) show that with St-orthonormal LDA and Sw-orthonormal LDA formulations, solutions to all four major LDA objective functions are identical. Furthermore, we perform an indepth analysis to show that the LDA sometimes performs poorly due to over-fitting, i.e., it picks up PCA dimensions with small eigenvalues. From this analysis, we propose a stable LDA which uses PCA first to reduce to a small PCA subspace and do LDA in the subspace.

References

[1]

Belhumeur, P.; Hespanha, J.; and Kriengman, D. 1997. Eigenfaces vs fisherfaces: recognition using class specific linear projection. IEEE Trans. Pattern Analysis and Machine Intelligence 19(7):711-720.

Crossref

Google Scholar

[2]

Chen, L.; Liao, H.; Lin, J.; Ko, M.; and Yu, G. 2000. A new lda-based face recognition system which can solve the small sample size problem. Pattern Recognition 33:17131726.

Google Scholar

[3]

Duda, R. O.; Hart, P. E.; and Stork, D. G. 2000. Pattern Classification, 2nd ed. Wiley.

Crossref

Google Scholar

[4]

Fisher, R. A. 1936. The use of multiple measurements in taxonomic problems. Annals of Eugenics 7.

Google Scholar

[5]

Fukunaga, K. 1990. Introduction to statistical pattern recognition. Academic Press Professional, 2nd edition.

Crossref

Google Scholar

[6]

Georghiades, A.; Belhumeur, P.; and Kriegman, D. 2001. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intelligence 23(6):643-660.

Crossref

Google Scholar

[7]

Hastie, T.; Tibshirani, R.; and Friedman, J. 2001. Elements of Statistical Learning. Springer Verlag.

Google Scholar

[8]

Jin, Z.; Yang, J.; Hu, Z.; and Lou, Z. 2001. Face recognition based on the uncorrelated discriminant transformation. Pattern Recognition 34:14051416.

Google Scholar

[9]

LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86(11):2278-2324.

Google Scholar

[10]

Luo, D.; Ding, C.; and Huang, H. 2009. Symmetric two dimensional linear discriminant analysis (2DLDA). IEEE Conf. Computer Vision and Pattern Recognition.

Google Scholar

[11]

Park, H., and Howland, P. 2004. Generalizing discriminant analysis using the generalized singular value decomposition. IEEE. Trans. on Pattern Analysis and Machine Intelligence 26:995 - 1006.

Crossref

Google Scholar

[12]

Turk, M. A., and Pentland, A. P. 1991. Face recognition using eigenfaces. In IEEE Conference on Computer Vision and Pattern Recognition, 586-591.

Google Scholar

[13]

Ye, J., and Xiong, T. 2006. Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. J. Machine Learning Research 7:1183-1204.

Crossref

Google Scholar

[14]

Ye, J.; Janardan, R.; Li, Q.; et al. 2004. Two-dimensional linear discriminant analysis. Advances in Neural Information Processing Systems 17:1569-1576.

Google Scholar

Cited By

View all

Fang YLi YLei CLi YDeng X(2018)Hypergraph expressing low-rank feature selection algorithmMultimedia Tools and Applications10.5555/3288251.328828977:22(29551-29572)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.5555/3288251.3288289
Laohakiat SPhimoltares SLursinsap C(2017)A clustering algorithm for stream data with LDA-based unsupervised localized dimension reductionInformation Sciences: an International Journal10.1016/j.ins.2016.11.018381:C(104-123)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1016/j.ins.2016.11.018
Hu RCheng DHe WWen GZhu YZhang JZhang S(2017)Low-rank feature selection for multi-view regressionMultimedia Tools and Applications10.1007/s11042-016-4119-276:16(17479-17495)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s11042-016-4119-2

Linear discriminant analysis: new formulations and overfit analysis
1. Computing methodologies

Recommendations

Graph regularized linear discriminant analysis and its generalization

Linear discriminant analysis (LDA) is a powerful dimensionality reduction technique, which has been widely used in many applications. Although, LDA is well-known for its discriminant capability, it clearly does not capture the geometric structure of the ...
Two-dimensional Linear Discriminant Analysis
NIPS'04: Proceedings of the 18th International Conference on Neural Information Processing Systems

Linear Discriminant Analysis (LDA) is a well-known scheme for feature extraction and dimension reduction. It has been used widely in many applications involving high-dimensional data, such as face recognition and image retrieval. An intrinsic limitation ...
Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis

Dimensionality reduction is an important pre-processing step in many applications. Linear discriminant analysis (LDA) is a classical statistical approach for supervised dimensionality reduction. It aims to maximize the ratio of the between-class ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

AAAI'11: Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence

August 2011

1883 pages

Publisher

AAAI Press

Publication History

Published: 07 August 2011

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Fang YLi YLei CLi YDeng X(2018)Hypergraph expressing low-rank feature selection algorithmMultimedia Tools and Applications10.5555/3288251.328828977:22(29551-29572)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.5555/3288251.3288289
Laohakiat SPhimoltares SLursinsap C(2017)A clustering algorithm for stream data with LDA-based unsupervised localized dimension reductionInformation Sciences: an International Journal10.1016/j.ins.2016.11.018381:C(104-123)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1016/j.ins.2016.11.018
Hu RCheng DHe WWen GZhu YZhang JZhang S(2017)Low-rank feature selection for multi-view regressionMultimedia Tools and Applications10.1007/s11042-016-4119-276:16(17479-17495)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s11042-016-4119-2

Abstract

References

Cited By

Recommendations

Graph regularized linear discriminant analysis and its generalization

Two-dimensional Linear Discriminant Analysis

Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations