More Web Proxy on the site http://driver.im/

Article

Self-taught learning: transfer learning from unlabeled data

Authors:

Benjamin Packer,

Andrew Y. NgAuthors Info & Claims

ICML '07: Proceedings of the 24th international conference on Machine learning

Pages 759 - 766

https://doi.org/10.1145/1273496.1273592

Published: 20 June 2007 Publication History

Abstract

We present a new machine learning framework called "self-taught learning" for using unlabeled data in supervised classification tasks. We do not assume that the unlabeled data follows the same class labels or generative distribution as the labeled data. Thus, we would like to use a large number of unlabeled images (or audio samples, or text documents) randomly downloaded from the Internet to improve performance on a given image (or audio, or text) classification task. Such unlabeled data is significantly easier to obtain than in typical semi-supervised or transfer learning settings, making self-taught learning widely applicable to many practical learning problems. We describe an approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data. These features form a succinct input representation and significantly improve classification performance. When using an SVM for classification, we further show how a Fisher kernel can be learned for this representation.

References

[1]

Ando, R. K., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6, 1817--1853.

Digital Library

[2]

Baxter, J. (1997). Theoretical models of learning to learn. In T. Mitchell and S. Thrun (Eds.), Learning to learn.

Digital Library

[3]

Blei, D., Ng, A. Y., & Jordan, M. (2002). Latent dirichlet allocation. NIPS.

[4]

Caruana, R. (1997). Multitask learning. ML Journal, 28.

Digital Library

[5]

Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A. (1990). Indexing by latent semantic analysis. J. Am. Soc. Info. Sci., 41, 391--407.

[6]

Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. Ann. Stat., 32, 407--499.

[7]

Fei-Fei, L., Fergus, R., & Perona, P. (2004). Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. CVPR Workshop on Gen.-Model Based Vision.

Digital Library

[8]

Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.

[9]

Holub, A., Welling, M., & Perona, P. (2005). Combining generative models and Fisher kernels for object class recognition. ICCV.

Digital Library

[10]

Hoyer, P. O. (2004). Non-negative matrix factorization with sparseness constraints. JMLR, 5, 1457--1469.

Digital Library

[11]

Jaakkola, T., & Haussler, D. (1998). Exploiting generative models in discriminative classifiers. NIPS.

Digital Library

[12]

Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. CVPR.

Digital Library

[13]

Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007). Efficient sparse coding algorithms. NIPS.

[14]

Ng, A. Y. (2004). Feature selection, L ₁ vs. L ₂ regularization, and rotational invariance. ICML.

Digital Library

[15]

Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103--134.

Digital Library

[16]

Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607--609.

[17]

Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290.

[18]

Serre, T., Wolf, L., & Poggio, T. (2005). Object recognition with features inspired by visual cortex. CVPR.

Digital Library

[19]

Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319--2323.

[20]

Thrun, S. (1996). Is learning the n-th thing any easier than learning the first? NIPS.

[21]

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B., 58, 267--288.

[22]

Tsuda, K., Kin, T., & Asai, K. (2002). Marginalized kernels for biological sequences. Bioinformatics, 18.

[23]

Zhang, H., Berg, A., Maire, M., & Malik, J. (2006). SVMKNN: Discriminative nearest neighbor classification for visual category recognition. CVPR.

Digital Library

Cited By

Akaadom BAgyekum FForson J(2024)Moving from Conventional to Online Instruction: Students’ Perspectives from the Global COVID-19 LockdownEuropean Journal of Theoretical and Applied Sciences10.59324/ejtas.2024.2(2).312:2(353-368)Online publication date: 1-Mar-2024
https://doi.org/10.59324/ejtas.2024.2(2).31
El Moutaouakil KRoudani MOuhmid AZhilenkov AMobayen S(2024)Decomposition and Symmetric Kernel Deep Neural Network Fuzzy Support Vector MachineSymmetry10.3390/sym1612158516:12(1585)Online publication date: 27-Nov-2024
https://doi.org/10.3390/sym16121585
Zhang MGu XQi JZhang ZYang HXu JPeng CLi H(2024)CDEST: Class Distinguishability-Enhanced Self-Training Method for Adopting Pre-Trained Models to Downstream Remote Sensing Image Semantic SegmentationRemote Sensing10.3390/rs1607129316:7(1293)Online publication date: 6-Apr-2024
https://doi.org/10.3390/rs16071293
Show More Cited By

Self-taught learning: transfer learning from unlabeled data
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Self-taught clustering
ICML '08: Proceedings of the 25th international conference on Machine learning

This paper focuses on a new clustering task, called self-taught clustering. Self-taught clustering is an instance of unsupervised transfer learning, which aims at clustering a small collection of target unlabeled data with the help of a large amount of ...
Supervised self-taught learning: actively transferring knowledge from unlabeled data
IJCNN'09: Proceedings of the 2009 international joint conference on Neural Networks

We consider the task of Self-taught Learning (STL) from unlabeled data. In contrast to semi-supervised learning, which requires unlabeled data to have the same set of class labels as labeled data, STL can transfer knowledge from different types of ...
Self-taught learning

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '07: Proceedings of the 24th international conference on Machine learning

June 2007

1233 pages

ISBN:9781595937933

DOI:10.1145/1273496

Editor:
Zoubin Ghahramani
University of Cambridge, United Kingdom

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Machine Learning Journal

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

ICML '07 & ILP '07

Sponsor:

ICML '07 & ILP '07: The 24th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 20 - 24, 2007

Oregon, Corvalis, USA

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

990
Total Citations
View Citations
10,721
Total Downloads

Downloads (Last 12 months)273
Downloads (Last 6 weeks)37

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Akaadom BAgyekum FForson J(2024)Moving from Conventional to Online Instruction: Students’ Perspectives from the Global COVID-19 LockdownEuropean Journal of Theoretical and Applied Sciences10.59324/ejtas.2024.2(2).312:2(353-368)Online publication date: 1-Mar-2024
https://doi.org/10.59324/ejtas.2024.2(2).31
El Moutaouakil KRoudani MOuhmid AZhilenkov AMobayen S(2024)Decomposition and Symmetric Kernel Deep Neural Network Fuzzy Support Vector MachineSymmetry10.3390/sym1612158516:12(1585)Online publication date: 27-Nov-2024
https://doi.org/10.3390/sym16121585
Zhang MGu XQi JZhang ZYang HXu JPeng CLi H(2024)CDEST: Class Distinguishability-Enhanced Self-Training Method for Adopting Pre-Trained Models to Downstream Remote Sensing Image Semantic SegmentationRemote Sensing10.3390/rs1607129316:7(1293)Online publication date: 6-Apr-2024
https://doi.org/10.3390/rs16071293
Chen HWang TKonpang JSirikham A(2024)Learning A-Share Stock Recommendation from Stock Graph and Historical Price SimultaneouslyElectronics10.3390/electronics1322442713:22(4427)Online publication date: 12-Nov-2024
https://doi.org/10.3390/electronics13224427
Yu CChen THsu CCheng H(2024)Incremental Scene Classification Using Dual Knowledge Distillation and Classifier Discrepancy on Natural and Remote Sensing ImagesElectronics10.3390/electronics1303058313:3(583)Online publication date: 31-Jan-2024
https://doi.org/10.3390/electronics13030583
Lu BLi QLiang Y(2024)A weakly supervised method for 3D object detection with partially annotated samplesMeasurement and Control10.1177/00202940241297568Online publication date: 28-Nov-2024
https://doi.org/10.1177/00202940241297568
He ZTu TWang KLuo BCheng DHu C(2024)Federated Spectrum Management Through Hedonic Coalition FormationProceedings of the 8th Asia-Pacific Workshop on Networking10.1145/3663408.3665823(217-218)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.1145/3663408.3665823
Chiang CSchwedt TDumkrieger GWang LChao COuellette HBanerjee IChen YJones BBurke KWang HMurray AMontenegro MStern JWhealy MKissoon NCutrer F(2024)Advancing toward precision migraine treatment: Predicting responses to preventive medications with machine learning models based on patient and migraine featuresHeadache: The Journal of Head and Face Pain10.1111/head.1480664:9(1094-1108)Online publication date: 23-Aug-2024
https://doi.org/10.1111/head.14806
Hu CTu TGong YJiang JZheng ZCheng D(2024)Tackling Multiplayer Interaction for Federated Generative Adversarial NetworksIEEE Transactions on Mobile Computing10.1109/TMC.2024.343814823:12(14017-14030)Online publication date: Dec-2024
https://doi.org/10.1109/TMC.2024.3438148
Wang KLei XZhou WCheng SLi J(2024)Historical Information-Aided Monitoring of Few-Sample Modes in Industrial Processes With Orthogonal Transferred ProjectionIEEE Transactions on Industrial Informatics10.1109/TII.2024.339655120:8(10375-10386)Online publication date: Aug-2024
https://doi.org/10.1109/TII.2024.3396551
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents