More Web Proxy on the site http://driver.im/

Article

Self-taught learning: transfer learning from unlabeled data

Authors:

Benjamin Packer,

Andrew Y. NgAuthors Info & Claims

ICML '07: Proceedings of the 24th international conference on Machine learning

Pages 759 - 766

https://doi.org/10.1145/1273496.1273592

Published: 20 June 2007 Publication History

Abstract

We present a new machine learning framework called "self-taught learning" for using unlabeled data in supervised classification tasks. We do not assume that the unlabeled data follows the same class labels or generative distribution as the labeled data. Thus, we would like to use a large number of unlabeled images (or audio samples, or text documents) randomly downloaded from the Internet to improve performance on a given image (or audio, or text) classification task. Such unlabeled data is significantly easier to obtain than in typical semi-supervised or transfer learning settings, making self-taught learning widely applicable to many practical learning problems. We describe an approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data. These features form a succinct input representation and significantly improve classification performance. When using an SVM for classification, we further show how a Fisher kernel can be learned for this representation.

References

[1]

Ando, R. K., & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. JMLR, 6, 1817--1853.

Digital Library

[2]

Baxter, J. (1997). Theoretical models of learning to learn. In T. Mitchell and S. Thrun (Eds.), Learning to learn.

Digital Library

[3]

Blei, D., Ng, A. Y., & Jordan, M. (2002). Latent dirichlet allocation. NIPS.

[4]

Caruana, R. (1997). Multitask learning. ML Journal, 28.

Digital Library

[5]

Deerwester, S. C., Dumais, S. T., Landauer, T. K., Furnas, G. W., & Harshman, R. A. (1990). Indexing by latent semantic analysis. J. Am. Soc. Info. Sci., 41, 391--407.

[6]

Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. Ann. Stat., 32, 407--499.

[7]

Fei-Fei, L., Fergus, R., & Perona, P. (2004). Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories. CVPR Workshop on Gen.-Model Based Vision.

Digital Library

[8]

Hinton, G. E., & Salakhutdinov, R. R. (2006). Reducing the dimensionality of data with neural networks. Science, 313, 504--507.

[9]

Holub, A., Welling, M., & Perona, P. (2005). Combining generative models and Fisher kernels for object class recognition. ICCV.

Digital Library

[10]

Hoyer, P. O. (2004). Non-negative matrix factorization with sparseness constraints. JMLR, 5, 1457--1469.

Digital Library

[11]

Jaakkola, T., & Haussler, D. (1998). Exploiting generative models in discriminative classifiers. NIPS.

Digital Library

[12]

Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. CVPR.

Digital Library

[13]

Lee, H., Battle, A., Raina, R., & Ng, A. Y. (2007). Efficient sparse coding algorithms. NIPS.

[14]

Ng, A. Y. (2004). Feature selection, L ₁ vs. L ₂ regularization, and rotational invariance. ICML.

Digital Library

[15]

Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103--134.

Digital Library

[16]

Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607--609.

[17]

Roweis, S. T., & Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290.

[18]

Serre, T., Wolf, L., & Poggio, T. (2005). Object recognition with features inspired by visual cortex. CVPR.

Digital Library

[19]

Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319--2323.

[20]

Thrun, S. (1996). Is learning the n-th thing any easier than learning the first? NIPS.

[21]

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B., 58, 267--288.

[22]

Tsuda, K., Kin, T., & Asai, K. (2002). Marginalized kernels for biological sequences. Bioinformatics, 18.

[23]

Zhang, H., Berg, A., Maire, M., & Malik, J. (2006). SVMKNN: Discriminative nearest neighbor classification for visual category recognition. CVPR.

Digital Library

Cited By

Obuchowicz RLasek JWodziński MPiórkowski AStrzelecki MNurzynska K(2025)Artificial Intelligence-Empowered Radiology—Current Status and Critical ReviewDiagnostics10.3390/diagnostics1503028215:3(282)Online publication date: 24-Jan-2025
https://doi.org/10.3390/diagnostics15030282
Thukral MDhekane SHiremath SHaresamudram HPloetz T(2025)Layout-Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST)Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/37122789:1(1-38)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3712278
Tavanti ENepa PGabbrielli RPirozzi M(2025)Review on Systems Combining Computer Vision and Radio Frequency IdentificationIEEE Internet of Things Journal10.1109/JIOT.2024.348475512:2(1291-1319)Online publication date: 15-Jan-2025
https://doi.org/10.1109/JIOT.2024.3484755
Show More Cited By

Self-taught learning: transfer learning from unlabeled data
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches

Recommendations

Self-taught clustering
ICML '08: Proceedings of the 25th international conference on Machine learning

This paper focuses on a new clustering task, called self-taught clustering. Self-taught clustering is an instance of unsupervised transfer learning, which aims at clustering a small collection of target unlabeled data with the help of a large amount of ...
Supervised self-taught learning: actively transferring knowledge from unlabeled data
IJCNN'09: Proceedings of the 2009 international joint conference on Neural Networks

We consider the task of Self-taught Learning (STL) from unlabeled data. In contrast to semi-supervised learning, which requires unlabeled data to have the same set of class labels as labeled data, STL can transfer knowledge from different types of ...
Self-taught learning

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICML '07: Proceedings of the 24th international conference on Machine learning

June 2007

1233 pages

ISBN:9781595937933

DOI:10.1145/1273496

Editor:
Zoubin Ghahramani
University of Cambridge, United Kingdom

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Machine Learning Journal

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

ICML '07 & ILP '07

Sponsor:

ICML '07 & ILP '07: The 24th Annual International Conference on Machine Learning held in conjunction with the 2007 International Conference on Inductive Logic Programming

June 20 - 24, 2007

Oregon, Corvalis, USA

Acceptance Rates

Overall Acceptance Rate 140 of 548 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,023
Total Citations
View Citations
10,781
Total Downloads

Downloads (Last 12 months)249
Downloads (Last 6 weeks)35

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Obuchowicz RLasek JWodziński MPiórkowski AStrzelecki MNurzynska K(2025)Artificial Intelligence-Empowered Radiology—Current Status and Critical ReviewDiagnostics10.3390/diagnostics1503028215:3(282)Online publication date: 24-Jan-2025
https://doi.org/10.3390/diagnostics15030282
Thukral MDhekane SHiremath SHaresamudram HPloetz T(2025)Layout-Agnostic Human Activity Recognition in Smart Homes through Textual Descriptions Of Sensor Triggers (TDOST)Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/37122789:1(1-38)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3712278
Tavanti ENepa PGabbrielli RPirozzi M(2025)Review on Systems Combining Computer Vision and Radio Frequency IdentificationIEEE Internet of Things Journal10.1109/JIOT.2024.348475512:2(1291-1319)Online publication date: 15-Jan-2025
https://doi.org/10.1109/JIOT.2024.3484755
Tu THe ZZheng ZZheng ZJiang JGong YHu CCheng D(2025)Toward Lifelong Unseen Task Processing With a Lightweight Unlabeled Data Schema for AIoTIEEE Internet of Things Journal10.1109/JIOT.2024.339628212:4(3441-3452)Online publication date: 15-Feb-2025
https://doi.org/10.1109/JIOT.2024.3396282
Smiliotopoulos CKambourakis GKolias CGritzalis S(2025)Assessing the detection of lateral movement through unsupervised learning techniquesComputers and Security10.1016/j.cose.2024.104190149:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.cose.2024.104190
Faisal MMecheter IQiblawey YFernandez JChowdhury MKiranyaz S(2025)Deep learning in automated power line inspection: A reviewApplied Energy10.1016/j.apenergy.2025.125507385(125507)Online publication date: May-2025
https://doi.org/10.1016/j.apenergy.2025.125507
Gholizade MSoltanizadeh HRahmanimanesh MSana S(2025)A review of recent advances and strategies in transfer learningInternational Journal of System Assurance Engineering and Management10.1007/s13198-024-02684-2Online publication date: 21-Feb-2025
https://doi.org/10.1007/s13198-024-02684-2
Yaqoob MIshaq MAnsari MQaiser YHussain RRabbani HGarwood RSeers T(2025)Advancing paleontology: a survey on deep learning methodologies in fossil image analysisArtificial Intelligence Review10.1007/s10462-024-11080-y58:3Online publication date: 6-Jan-2025
https://doi.org/10.1007/s10462-024-11080-y
Akaadom BAgyekum FForson J(2024)Moving from Conventional to Online Instruction: Students’ Perspectives from the Global COVID-19 LockdownEuropean Journal of Theoretical and Applied Sciences10.59324/ejtas.2024.2(2).312:2(353-368)Online publication date: 1-Mar-2024
https://doi.org/10.59324/ejtas.2024.2(2).31
Hildebrand NKummer S(2024)Exploring Urban Environment Heterogeneity: Impact of Urban Sprawl on Charging Infrastructure Demand over TimeWorld Electric Vehicle Journal10.3390/wevj1512058915:12(589)Online publication date: 20-Dec-2024
https://doi.org/10.3390/wevj15120589
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten