Preserve Knowledge with Auxiliary Feature Extractor for Class Incremental Learning
Authors: 
Huihui Jie, Yuesheng ZhuAuthors Info & Claims
Pages 8 - 15
Abstract
Class incremental learning (CIL) aims to achieve the ability to learn knowledge from the data of novel classes that arrive incrementally. To this end, the exemplar-based method stores a small number of samples of old classes and has been proven to be effective yet it causes the severe data imbalance issue. An approach named SS-IL solves the issue effectively and achieves strong state-of-the-art on large-scale CIL benchmark datasets while behaving badly on small ones. In this paper, we observe that the poor performance of SS-IL on small datasets could stem from not fully stimulating the potentiality of the learned representation of old classes, especially the initial classes. We propose an auxiliary Weight Scaling Feature Extractor (aWSFE) to better maintain and exploit the essential semantics of old classes. This auxiliary extractor is used as a plug-in module with the main classification network based on SS-IL in parallel. We perform a special design for the two branches so that the feature vectors from the main and auxiliary extractor can be integrated easily without an additional aggregation process. After obtaining the updated representations, we finetuning the classifier based on a balanced subset of training data to further promote performance. We conduct extensive experiments on two small-scale CIL benchmark datasets: CIFAR-100 and ImageNet-Sub. Results show that the proposed method effectively alleviates the forgetting of old knowledge and significantly improves the performance of SS-IL on small datasets.
References
[1]
W. Liu, Z. Wang, X. Liu, N. Zeng, Y. Liu, and F. Alsaadi, “A survey of deep neural network architectures and their applications,” in Neurocomputing, vol. 234, pp. 11-26, 2017.
[2]
A. C, A. P, and E. C, “An analysis of deep neural network models for practical applications,” arXiv preprint arXiv:1605.07678, 2016.
[3]
S. Hu, Y. Zuo, L. Wang, and P. Liu, "A Review about Building Hidden Layer Methods of Deep Learning," in Journal of Advances in Information Technology, vol. 7, pp. 13-22, 2016.
[4]
M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” in Psychology of learning and motivation, vol. 24, pp. 109-165, 1989.
[5]
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” arXiv preprint arXiv:1312.6211, 2013.
[6]
F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in International Conference on Machine Learning, pp. 3987-3995, 2017.
[7]
A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr, “Riemannian walk for incremental learning: Understanding forgetting and intransigence,” in Proceedings of the European Conference on Computer Vision, pp. 532-547, 2018.
[8]
J. Serra, D. Suris, M. Miron, and A. Karatzoglou, “Overcoming catastrophic forgetting with hard attention to the task,” in International Conference on Machine Learning, pp. 4548-4557, 2018.
[9]
D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
[10]
Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12): 2935-2947, 2017.
[11]
H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” in Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
[12]
J. Serra, D. Suris, M. Miron, and A. Karatzoglou, “Overcoming catastrophic forgetting with hard attention to the task,” in International Conference on Machine Learning, pp. 4548-4557, 2018.
[13]
R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in Proceedings of the European Conference on Computer Vision, pp. 144-161, 2018.
[14]
G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in Advances in Neural Information Processing Systems Workshop, 2015.
[15]
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
[16]
F.M. Castro, M. J. Marín-Jiménez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in Proceedings of the European Conference on Computer Vision, pp. 233-248, 2018.
[17]
E. Belouadah and A. Popescu, “Il2m: Class incremental learning with dual memory,” in The IEEE International Conference on Computer Vision, pp. 583-592, 2019.
[18]
Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu, “Large scale incremental learning,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp.374-382, 2019.
[19]
Y. Liu, Y. Su, A. Liu, B. Schiele, and Q. Sun, “Mnemonics training: Multi-class incremental learning without forgetting,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12245-12254, 2020.
[20]
S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
[21]
H. Ahn, J. Kwak, S. Lim, H. Bang, H. Kim, and T. Moon, “SS-IL: Separated Softmax for Incremental Learning,” in The IEEE International Conference on Computer Vision, 2021.
[22]
B. Zhao, X. Xiao, G. Gan, B. Zhang, and S. Xia, “Maintaining discrimination and fairness in class incremental learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 13208–13217, 2020.
[23]
Y. Liu, B. Schiele, and Q. Sun, “Adaptive Aggregation Networks for Class-Incremental Learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2544-2553, 2021.
[24]
P. Dhar, R. V. Singh, K.C. Peng, Z. Wu, and R. Chellappa, “Learning Without Memorizing,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5133-5141, 2019.
[25]
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in The IEEE International Conference on Computer Vision, pp. 618-626, 2017.
[26]
A. Douillard, M. Cord, C. Ollion, T. Robert, and E.Valle. “PODNet: Pooled outputs distillation for small-tasks incremental learning,” in Proceedings of the European Conference on Computer Vision, 2020.
[27]
C. Simon, P. Koniusz, and M. Harandi, “On Learning the Geodesic Path for Incremental Learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[28]
B. Gong, Y. Shi, F. Sha, and K. Grauman, “Geodesic flow kernel for unsupervised domain adaptation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2066–2073, 2012.
[29]
J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong Learning with Dynamically Expandable Networks,” in International Conference on Learning Representations, 2018.
[30]
J. Rajasegaran, S. Khan, M. Hayat, F. S. Khan, and M. Shah, “iTAML: An Incremental Task-Agnostic Meta-learning Approach,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 13588-13597, 2021.
[31]
D. Abati, J. Tomczak, T. Blankevoort, S. Calderara, R. Cucchiara, and B. E. Bejnordi, “Conditional channel gated networks for task-aware continual learning,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3931-3940, 2020.
[32]
A. Krizhevsky, and G. Hinton. “Learning multiple layers of features from tiny images,” Citeseer, Technical Report, 2009.
[33]
J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and F. F. Li, “ImageNet: a Large-Scale Hierarchical Image Database,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 248-255, 2009.
[34]
X. Hu, K. Tang, C. Miao, X. Hua, and H. Zhang, “Distilling Causal Effect of Data in Class-Incremental Learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[35]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Index Terms
- Preserve Knowledge with Auxiliary Feature Extractor for Class Incremental Learning
Recommendations
Future-proofing class-incremental learning: Future-proofing class-incremental learning
AbstractExemplar-free class incremental learning is a highly challenging setting where replay memory is unavailable. Methods relying on frozen feature extractors have drawn attention recently in this setting due to their impressive performances and lower ...
Recognizing extended surrounding contexts via class incremental learning
UbiComp/ISWC'15 Adjunct: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable ComputersBenefit from widely used Bluetooth sensor, user surrounding contexts can be availably recognized leveraging Bluetooth data. Most existing studies seldom deal with newly extended surrounding contexts which results in degrading the recognition performance,...
Rebalancing network with knowledge stability for class incremental learning
AbstractClass incremental learning (CIL) has been proposed to solve the problem of learning to classify new classes while maintaining the performance on old classes. A typical strategy is to update the old classification model with entire new class data ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2022
241 pages
ISBN:9781450397315
DOI:10.1145/3561613
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 November 2022
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICCCV 2022
ICCCV 2022: 2022 The 5th International Conference on Control and Computer Vision
August 19 - 21, 2022
Xiamen, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 45Total Downloads
- Downloads (Last 12 months)20
- Downloads (Last 6 weeks)3
Reflects downloads up to 15 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format