Abstract
Recent advancements in deep learning algorithms have shown remarkable performance on trained tasks, yet they struggle with “catastrophic forgetting” when faced with new tasks, highlighting the need for Continual Learning (CL) methods that update models efficiently without losing prior knowledge. CL models, constrained by limited visibility of the dataset for each task, develop a significant dependency on past tasks, complicating the integration of new information and maintaining robustness against future tasks. This paper proposes a novel CL method that leverages contrastive learning to secure a latent space for future data representation, reducing the dependency on past tasks and enhancing model adaptability. By distinguishing class spaces in the latent domain and re-representing these as sets of means and variances, our method effectively preserves past knowledge while ensuring future robustness. Experimental results show our method surpasses existing CL methods by a significant margin, proving its efficacy in handling information across past, present, and future tasks, thus establishing a robust solution for the challenges of catastrophic forgetting and task dependency in CL.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Guo, Y., Liu, B., et al.: Online continual learning through mutual information maximization. In: ICML, pp. 8109–8126 (2022)
Zhao, Z., Zhang, Z., et al.: Rethinking gradient projection continual learning: stability/plasticity feature space decoupling. In: CVPR, pp. 3718–3727 (2023)
Buzzega, P., Boschini, M., et al.: Rethinking experience replay: a bag of tricks for continual learning. In: International Conference on Pattern Recognition, pp. 2180–2187 (2021)
Lomonaco, V., Maltoni, D., et al.: Rehearsal-free continual learning over small non-IID batches. In: CVPR Workshops (2020)
Prabhu, A., Torr, P.H.S., Dokania, P.K.: GDumb: a simple approach that questions our progress in continual learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 524–540. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_31
Chaudhry, A., Ranzato, M.A., et al.: Efficient lifelong learning with A-GEM. arXiv preprint arXiv:1812.00420 (2018)
Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.S.: Riemannian walk for incremental learning: understanding forgetting and intransigence. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 556–572. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_33
Yoon, J., Madaan, D., et al.: Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085 (2021)
Rebuffi, S.A., Kolesnikov, A., et al.: iCaRL: incremental classifier and representation learning. In: CVPR, pp. 2001–2010 (2017)
Javed, K., White, M.: Meta-learning representations for continual learning. In: NeuRIPS, vol. 32 (2019)
Gupta, G., Yadav, K., et al.: Look-ahead meta learning for continual learning. In: NeuRIPS, vol. 33, pp. 11588–11598 (2020)
Gallardo, J., Hayes, T.L., et al.: Self-supervised training enhances online continual learning. arXiv preprint arXiv:2103.14010 (2021)
Zhang, M., Wang, T., et al.: Variational prototype replays for continual learning, arXiv preprint arXiv:1905.09447 (2019)
Zhang, B., Guo, Y., et al.: Memory recall: a simple neural network training framework against catastrophic forgetting. IEEE Trans. Neural Netw. Learn. Syst. 33(5), 2010–2022 (2021)
Kirkpatrick, J., Pascanu, R., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
Zeng, G., Chen, Y., et al.: Continual learning of context-dependent processing in neural networks. Nat. Mach. Intell. 1(8), 364–372 (2019)
Kim, T.H., Moon, H.J., et al.: Gradient regularization with multivariate distribution of previous knowledge for continual learning. In: International Conference on Intelligent Data Engineering and Automated Learning, pp. 359–368 (2022)
Wang, Z., Liu, L., et al.: Continual learning with lifelong vision transformer. In: CVPR, pp. 171–181 (2022)
Riemer, M., Cases, I., et al.: Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.11910 (2018)
Lopez-Paz, D., Ranzato, M. A.: Gradient episodic memory for continual learning. In: NeuRIPS, vol. 30 (2017)
Aljundi, R., Lin, M., et al.: Gradient based sample selection for online continual learning. In: NeuRIPS, vol. 32 (2019)
Chen, D., Lin, Y., et al.: Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In: AAAI, vol. 34, no. 4, pp. 3438–3445 (2020)
Chaudhry, A., Gordo, A., et al.: Using hindsight to anchor past knowledge in continual learning. In: AAAI, vol. 35, no. 8, pp. 6993–7001 (2021)
Buzzega, P., Boschini, M., et al.: Rethinking experience replay: a bag of tricks for continual learning. In: CVPR, pp. 2180–2187 (2021)
Bang, J., Kim, H., et al.: Rainbow memory: continual learning with a memory of diverse samples. In: CVPR, pp. 8218–8227 (2021)
Buzzega, P., Boschini, M., et al.: Dark experience for general continual learning: a strong, simple baseline. In: NeuRIPS, vol. 33, pp. 15920–15930 (2020)
Acknowledgements
This work was supported by the Yonsei Fellow Program funded by Lee Youn Jae, IITP grant funded by the Korea government (MSIT) (No. 2022–0–00113, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework), and Air Force Defense Research Sciences Program funded by Air Force Office of Scientific Research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Moon, HJ., Cho, SB. (2024). Contrastive Learning of Multivariate Gaussian Distributions of Incremental Classes for Continual Learning. In: Ferrández Vicente, J.M., Val Calvo, M., Adeli, H. (eds) Artificial Intelligence for Neuroscience and Emotional Systems. IWINAC 2024. Lecture Notes in Computer Science, vol 14674. Springer, Cham. https://doi.org/10.1007/978-3-031-61140-7_49
Download citation
DOI: https://doi.org/10.1007/978-3-031-61140-7_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-61139-1
Online ISBN: 978-3-031-61140-7
eBook Packages: Computer ScienceComputer Science (R0)