More Web Proxy on the site http://driver.im/

Article

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning

Authors:

Arthur Douillard,

Charles Ollion,

Eduardo ValleAuthors Info & Claims

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX

Pages 86 - 102

https://doi.org/10.1007/978-3-030-58565-5_6

Published: 23 August 2020 Publication History

Abstract

Lifelong learning has attracted much attention, but existing works still struggle to fight catastrophic forgetting and accumulate knowledge over long stretches of incremental learning. In this work, we propose PODNet, a model inspired by representation learning. By carefully balancing the compromise between remembering the old classes and learning new ones, PODNet fights catastrophic forgetting, even over very long runs of small incremental tasks – a setting so far unexplored by current works. PODNet innovates on existing art with an efficient spatial-based distillation-loss applied throughout the model and a representation comprising multiple proxy vectors for each class. We validate those innovations thoroughly, comparing PODNet with three state-of-the-art models on three datasets: CIFAR100, ImageNet100, and ImageNet1000. Our results showcase a significant advantage of PODNet over existing art, with accuracy gains of 12.10, 6.51, and 2.85 percentage points, respectively.

References

[1]

Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV) (2018)

[2]

Allen, K., Shelhamer, E., Shin, H., Tenenbaum, J.: Infinite mixture prototypes for few-shot learning. In: International Conference on Machine Learning (ICML) (2019)

[3]

Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV) (2018)

[4]

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

[5]

Dhar, P., Singh, R.V., Peng, K.C., Wu, Z., Chellappa, R.: Learning without memorizing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

[6]

Fernando, C., et al.: PathNet: evolution channels gradient descent in super neural networks. arXiv preprint library (2017)

[7]

French R Catastrophic forgetting in connectionist networks Trends Cogn. Sci. 1999 3 4 128-135

[8]

Gidaris, S., Komodakis, N.: Dynamic few-shot visual learning without forgetting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

[9]

Goldberger, J., Hinton, G.E., Roweis, S.T., Salakhutdinov, R.R.: Neighbourhood components analysis. In: Advances in Neural Information Processing Systems (NeurIPS) (2005)

[10]

Golkar, S., Kagan, M., Cho, K.: Continual learning via neural pruning. In: Advances in Neural Information Processing Systems (NeurIPS), Neuro AI Workshop (2019)

[11]

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

[12]

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: Advances in Neural Information Processing Systems (NeurIPS), Deep Learning and Representation Learning Workshop (2015)

[13]

Hou, S., Pan, X., Change Loy, C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

[14]

Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV) (2016)

[15]

Kemker, R., Kanan, C.: FearNet: Brain-inspired model for incremental learning. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

[16]

Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. In: Proceedings of the National Academy of Sciences (2017)

[17]

Komodakis, N., Zagoruyko, S.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: Proceedings of the International Conference on Learning Representations (ICLR) (2017)

[18]

Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report (2009)

[19]

Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. Object Categorization: Computer and Human Vision Perspectives. Cambridge University Press (2006)

[20]

Li, X., Zhou, Y., Wu, T., Socher, R., Xiong, C.: Learn to grow: a continual structure learning framework for overcoming catastrophic forgetting (2019)

[21]

Li, Z., Hoiem, D.: Learning without forgetting. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV) (2016)

[22]

Lomonaco, V., Maltoni, D.: CORe50: a new dataset and benchmark for continuous object recognition. In: Annual Conference on Robot Learning (2017)

[23]

Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems (NeurIPS) (2017)

[24]

Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (1999)

[25]

Luo, C., Zhan, J., Xue, X., Wang, L., Ren, R., Yang, Q.: Cosine normalization: Using cosine similarity instead of dot product in neural networks. In: International Conference on Artificial Neural Networks (2018)

[26]

Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

[27]

Paszke, A., et al.: Automatic differentiation in PyTorch. In: Advances in Neural Information Processing Systems (NeurIPS), Autodiff Workshop (2017)

[28]

Qi, H., Brown, M., Lowe, D.G.: Low-shot learning with imprinted weights. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

[29]

Qian, Q., Shang, L., Sun, B., Hu, J., Li, H., Jin, R.: SoftTriple loss: deep metric learning without triplet sampling. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2019)

[30]

Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

[31]

Robins A Catastrophic forgetting, rehearsal and pseudorehearsal Connection Sci. 1995 7 123-146

[32]

Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)

[33]

Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)

[34]

Thrun S Thrun S and Pratt L Lifelong learning algorithms Learning to Learn 1998 Boston, MA Springer 181-209

[35]

Wu, Y., et al.: Large scale incremental learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

[36]

Yoon, J., Yang, E., Lee, J., Hwang, S.J.: Lifelong learning with dynamically expandable networks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

[37]

Zhou, P., Mai, L., Zhang, J., Xu, N., Wu, Z., Davis, L.S.: M2KD: multi-model and multi-level knowledge distillation for incremental learning. arXiv preprint library (2019)

Cited By

Yang YZhou JDing XHuai TLiu SChen QXie YHe L(2025)Recent Advances of Foundation Language Models-based Continual Learning: A SurveyACM Computing Surveys10.1145/370572557:5(1-38)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3705725
Zheng BZhou DYe HZhan DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Multi-layer rehearsal feature augmentation for class-incremental learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694619(61649-61663)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694619
Yao XWang YZhu PLin WLi JLi WHu QSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Socialized learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694419(56927-56945)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694419
Show More Cited By

Index Terms

PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Multiple incremental decremental learning of support vector machines

We propose a multiple incremental decremental algorithm of support vector machines (SVM). In online learning, we need to update the trained model when some new observations arrive and/or some observations become obsolete. If we want to add or remove ...
Incremental Learning for Malware Classification in Small Datasets

Information security is an important research area. As a very special yet important case, malware classification plays an important role in information security. In the real world, the malware datasets are open-ended and dynamic, and new malware ...
An incremental probabilistic neural network for regression and reinforcement learning tasks
ICANN'10: Proceedings of the 20th international conference on Artificial neural networks: Part II

This paper presents a new probabilistic neural network model, called IPNN (for Incremental Probabilistic Neural Network), which is able to learn continuously probability distributions from data flows. The proposed model is inspired by the Specht's ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX

Aug 2020

838 pages

ISBN:978-3-030-58564-8

DOI:10.1007/978-3-030-58565-5

Editors:
Andrea Vedaldi
University of Oxford, Oxford, UK
,
Horst Bischof
Graz University of Technology, Graz, Austria
,
Thomas Brox
University of Freiburg, Freiburg im Breisgau, Germany
,
Jan-Michael Frahm
University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

© Springer Nature Switzerland AG 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 23 August 2020

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

86
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang YZhou JDing XHuai TLiu SChen QXie YHe L(2025)Recent Advances of Foundation Language Models-based Continual Learning: A SurveyACM Computing Surveys10.1145/370572557:5(1-38)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3705725
Zheng BZhou DYe HZhan DSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Multi-layer rehearsal feature augmentation for class-incremental learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694619(61649-61663)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694619
Yao XWang YZhu PLin WLi JLi WHu QSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Socialized learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694419(56927-56945)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694419
Wang RHwang JBoopathy AFiete ISalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Rapid learning without catastrophic forgetting in the Morris water mazeProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694144(50669-50682)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3694144
Piao HWu YWu DWei YSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Federated continual learning via prompt-based dual knowledge transferProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693723(40725-40739)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693723
Michel NWang MXiao LYamasaki TSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Rethinking momentum knowledge distillation in online continual learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693520(35607-35622)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693520
Lin WChen JHuang RDing HSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)An effective dynamic gradient calibration method for continual learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693273(29872-29889)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693273
Li DWang TChen JDai WZeng ZSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Harnessing neural unit dynamics for effective and scalable class-incremental learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693223(28688-28705)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693223
Cha SCho KMoon TSalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Regularizing with pseudo-negatives for continual self-supervised learningProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3692303(6048-6065)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3692303
Wang SZhan YLuo YHu HYu WWen YTao DLarson K(2024)Joint input and output coordination for class-incremental learningProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/565(5108-5116)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/565
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents