Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning
Authors: 
Anton Lee, Yaqian Zhang, 
Heitor Murilo Gomes, 
Albert Bifet, Bernhard PfahringerAuthors Info & Claims
Pages 4038 - 4042
Abstract
Continual learning aims to create artificial neural networks capable of accumulating knowledge and skills through incremental training on a sequence of tasks. The main challenge of continual learning is catastrophic interference, wherein new knowledge overrides or interferes with past knowledge, leading to forgetting. An associated issue is the problem of learning "cross-task knowledge," where models fail to acquire and retain knowledge that helps differentiate classes across task boundaries. A common solution to both problems is "replay," where a limited buffer of past instances is utilized to learn cross-task knowledge and mitigate catastrophic interference. However, a notable drawback of these methods is their tendency to overfit the limited replay buffer. In contrast, our proposed solution, SurpriseNet, addresses catastrophic interference by employing a parameter isolation method and learning cross-task knowledge using an auto-encoder inspired by anomaly detection. SurpriseNet is applicable to both structured and unstructured data, as it does not rely on image-specific inductive biases. We have conducted empirical experiments demonstrating the strengths of SurpriseNet on various traditional vision continual-learning benchmarks, as well as on structured data datasets. Source code made available at https://doi.org/10.5281/zenodo.8247906 and https://github.com/tachyonicClock/SurpriseNet-CIKM-23
Supplementary Material
Video presentation of "Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning"
- Download
- 61.82 MB
References
[1]
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. 2020. Dark Experience for General Continual Learning: a Strong, Simple Baseline. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/b704ea2c39778f07c617f6b7ce480e9e-Abstract.html
[2]
Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2019. Efficient Lifelong Learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6--9, 2019. OpenReview.net. https://openreview.net/forum?id=Hkf2_sC5FX
[3]
Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Trö ster, José del R. Millá n, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognit. Lett., Vol. 34, 15 (2013), 2033--2042. https://doi.org/10.1016/j.patrec.2012.12.014
[4]
M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021), 1--1. https://doi.org/10.1109/TPAMI.2021.3057446
[5]
Sebastian Farquhar and Yarin Gal. 2019. Towards Robust Evaluations of Continual Learning. arXiv:1805.09733 (Jun 2019). http://arxiv.org/abs/1805.09733 arXiv:1805.09733 [cs, stat].
[6]
Robert M French. [n.,d.]. Catastrophic Forgetting in Connectionist Networks., Vol. 3, 4 ( [n.,d.]), 8.
[7]
Jakob Drachmann Havtorn, Jes Frellsen, Søren Hauberg, and Lars Maaløe. 2021. Hierarchical VAEs Know What They Don't Know. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18--24 July 2021, Virtual Event (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 4117--4128. http://proceedings.mlr.press/v139/havtorn21a.html
[8]
Saurav Jha, Martin Schiemer, and Juan Ye. 2020. Continual Learning in Human Activity Recognition: an Empirical Analysis of Regularization. CoRR, Vol. abs/2007.03032 (2020). showeprint[arXiv]2007.03032 https://arxiv.org/abs/2007.03032
[9]
Saurav Jha, Martin Schiemer, Franco Zambonelli, and Juan Ye. 2021. Continual learning in sensor-based human activity recognition: An empirical benchmark analysis. Inf. Sci., Vol. 575 (2021), 1--21. https://doi.org/10.1016/j.ins.2021.04.062
[10]
Gyuhak Kim, Sepideh Esmaeilpour, Changnan Xiao, and Bing Liu. 2022a. Continual Learning Based on OOD Detection and Task Masking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19--20, 2022. IEEE, 3855--3865. https://doi.org/10.1109/CVPRW56347.2022.00431
[11]
Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, and Bing Liu. 2022b. A Theoretical Study on Solving Continual Learning. CoRR, Vol. abs/2211.02633 (2022). https://doi.org/10.48550/arXiv.2211.02633 showeprint[arXiv]2211.02633
[12]
Jong-Yeong Kim and Dong-Wan Choi. 2021. Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 9 (May 2021), 8137--8145. https://doi.org/10.1609/aaai.v35i9.16991
[13]
Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. CoRR, Vol. abs/1312.6114 (2013).
[14]
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2016. Overcoming catastrophic forgetting in neural networks. CoRR, Vol. abs/1612.00796 (2016). showeprint[arXiv]1612.00796 http://arxiv.org/abs/1612.00796
[15]
Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. (2009), 60.
[16]
Clayton Frederick Souza Leite and Yu Xiao. 2022. Resource-Efficient Continual Learning for Sensor-Based Human Activity Recognition. ACM Trans. Embed. Comput. Syst., Vol. 21, 6 (2022), 85:1--85:25. https://doi.org/10.1145/3530910
[17]
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, Vol. 40, 12 (2017), 2935--2947.
[18]
Arun Mallya and Svetlana Lazebnik. 2018. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, 7765--7773. https://doi.org/10.1109/CVPR.2018.00810
[19]
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. 2022. Class-Incremental Learning: Survey and Performance Evaluation on Image Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022), 1--20. https://doi.org/10.1109/TPAMI.2022.3213473
[20]
Michael McCloskey and Neal J. Cohen. 1989. Catastrophic Interference in Connectionist Networks : The Sequential Learning Problem. Psychology of Learning and Motivation, Vol. 24 (1989), 109--165. https://doi.org/10.1016/S0079--7421(08)60536--8
[21]
Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh. 2023. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, Vol. 160 (2023), 306--336. https://doi.org/10.1016/j.neunet.2023.01.014
[22]
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. 2018. Do Deep Generative Models Know What They Don't Know? (Oct 2018). https://arxiv.org/abs/1810.09136v3
[23]
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. 2022. Deep Learning for Anomaly Detection: A Review. Comput. Surveys, Vol. 54, 2 (Mar 2022), 1--38. https://doi.org/10.1145/3439950
[24]
German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, Vol. 113 (May 2019), 54--71. https://doi.org/10.1016/j.neunet.2019.01.012
[25]
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. iCaRL: Incremental Classifier and Representation Learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21--26, 2017. IEEE Computer Society, 5533--5542. https://doi.org/10.1109/CVPR.2017.587
[26]
Attila Reiss and Didier Stricker. 2012. Introducing a New Benchmarked Dataset for Activity Monitoring. In 16th International Symposium on Wearable Computers, ISWC 2012, Newcastle, United Kingdom, June 18--22, 2012. IEEE Computer Society, 108--109. https://doi.org/10.1109/ISWC.2012.13
[27]
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018. Overcoming Catastrophic Forgetting with Hard Attention to the Task. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm"a ssan, Stockholm, Sweden, July 10--15, 2018 (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 4555--4564. http://proceedings.mlr.press/v80/serra18a.html
[28]
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. 2020. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, Vol. 11 (2020), 4069.
[29]
Gido M. van de Ven and Andreas S. Tolias. 2019. Three scenarios for continual learning. CoRR, Vol. abs/1904.07734 (2019). showeprint[arXiv]1904.07734 http://arxiv.org/abs/1904.07734
[30]
Jindong Wang, Yiqiang Chen, Lisha Hu, Xiaohui Peng, and Philip S. Yu. 2018. Stratified Transfer Learning for Cross-domain Activity Recognition. In 2018 IEEE International Conference on Pervasive Computing and Communications, PerCom 2018, Athens, Greece, March 19--23, 2018. IEEE Computer Society, 1--10. https://doi.org/10.1109/PERCOM.2018.8444572
[31]
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. showeprint[arXiv]cs.LG/1708.07747 [cs.LG]
[32]
Juan Ye, Pakawat Nakwijit, Martin Schiemer, Saurav Jha, and Franco Zambonelli. 2021. Continual Activity Recognition with Generative Adversarial Networks. ACM Trans. Internet Things, Vol. 2, 2 (2021), 9:1--9:25. https://doi.org/10.1145/3440036
Index Terms
- Look At Me, No Replay! SurpriseNet: Anomaly Detection Inspired Class Incremental Learning
Recommendations
Continuous image anomaly detection based on contrastive lifelong learning
AbstractWith the development of deep learning techniques, an increasing number of anomaly detection methods based on deep neural networks have been proposed during the last decade. Nevertheless, these methods often suffer from catastrophic forgetting when ...
Sequential anomaly detection based on temporal-difference learning: Principles, models and case studies
Anomaly detection is an important problem that has been popularly researched within diverse research areas and application domains. One of the open problems in anomaly detection is the modeling and prediction of complex sequential data, which consist of ...
Bayesian Statistical Inference in Machine Learning Anomaly Detection
ICCIIS '10: Proceedings of the 2010 International Conference on Communications and Intelligence Information SecurityIntrusion Detection is an important component of the security of the Network, through the critical information of the network and host system, it can determine the user’s invasion of illegal and legal acts of the user misuse of resources, and make an ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2023
5508 pages
ISBN:9798400701245
DOI:10.1145/3583780
- General Chairs:
- Ingo Frommholz,
- Frank Hopfgartner,
- Mark Lee,
- Michael Oakes,
- Program Chairs:
- Mounia Lalmas,
- Min Zhang,
- Rodrygo Santos
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 October 2023
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
- NZ MBIE (Ministry of Business, Innovation and Employment)
Conference
CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management
October 21 - 25, 2023
Birmingham, United Kingdom
Acceptance Rates
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 207Total Downloads
- Downloads (Last 12 months)87
- Downloads (Last 6 weeks)3
Reflects downloads up to 14 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in