Abstract
We discuss a general formulation for the Continual Learning (CL) problem for classification—a learning task where a stream provides samples to a learner and the goal of the learner, depending on the samples it receives, is to continually upgrade its knowledge about the old classes and learn new ones. Our formulation takes inspiration from the open-set recognition problem where test scenarios do not necessarily belong to the training distribution. We also discuss various quirks and assumptions encoded in recently proposed approaches for CL. We argue that some oversimplify the problem to an extent that leaves it with very little practical importance, and makes it extremely easy to perform well on. To validate this, we propose GDumb that (1) greedily stores samples in memory as they come and; (2) at test time, trains a model from scratch using samples only in the memory. We show that even though GDumb is not specifically designed for CL problems, it obtains state-of-the-art accuracies (often with large margins) in almost all the experiments when compared to a multitude of recently proposed algorithms. Surprisingly, it outperforms approaches in CL formulations for which they were specifically designed. This, we believe, raises concerns regarding our progress in CL for classification. Overall, we hope our formulation, characterizations and discussions will help in designing realistically useful CL algorithms, and GDumb will serve as a strong contender for the same.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of Learning and Motivation (1989)
Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013)
Li, Z., Hoiem, D.: Learning without forgetting. TPAMI 40(12), 2935–2947 (2017)
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: CVPR (2017)
Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. ICML 70, 3987 (2017)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. PNAS 114(13), 3521–3526 (2017)
Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: NeurIP (2017)
Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: understanding forgetting and intransigence. In: ECCV (2018)
De Lange, M., et al.: Continual learning: a comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383 (2019)
Scheirer, W., Rocha, A., Sapkota, A., Boult, T.: Towards open set recognition. TPAMI 35(7), 1757–1772 (2012)
Aljundi, R., Caccia, L., Belilovsky, E., Caccia, M., Charlin, L., Tuytelaars, T.: Online continual learning with maximally interfered retrieval. In: NeurIPS (2019)
Jin, X., Du, J., Ren, X.: Gradient based memory editing for task-free continual learning (2020)
Dhar, P., Vikram Singh, R., Peng, K.C., Wu, Z., Chellappa, R.: Learning without memorizing. In: CVPR (2019)
Zhang, J., et al.: Class-incremental learning via deep model consolidation. In: WACV (2020)
Yu, L., et al.: Semantic drift compensation for class-incremental learning. In: CVPR (2020)
Wu, Y., et al.: Large scale incremental learning. In: CVPR (2019)
Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: CVPR (2019)
Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: ECCV (2018)
Belouadah, E., Popescu, A.: Il2m: class incremental learning with dual memory. In: ICCV (2019)
Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.: Maintaining discrimination and fairness in class incremental learning. In: CVPR (2020)
Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Small-task incremental learning. ECCV (2020)
Liu, Y., Su, Y., Liu, A.A., Schiele, B., Sun, Q.: Mnemonics training: multi-class incremental learning without forgetting. In: CVPR (2020)
Rajasegaran, J., Hayat, M., Khan, S., Khan, F.S., Shao, L.: Random path selection for incremental learning. In: NeurIPS (2019)
Rajasegaran, J., Khan, S., Hayat, M., Khan, F.S., Shah, M.: itaml: an incremental task-agnostic meta-learning approach. In: CVPR (2020)
Abati, D., Tomczak, J., Blankevoort, T., Calderara, S., Cucchiara, R., Bejnordi, B.E.: Conditional channel gated networks for task-aware continual learning. In: CVPR (2020)
Rusu, A.A., et al.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
Yoon, J., Lee, J., Yang, E., Hwang, S.J.: Lifelong learning with dynamically expandable network. In: ICLR (2018)
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NeurIPS (2017)
Schwarz, J., et al.: Progress & compress: a scalable framework for continual learning. ICML (2018)
Yoon, J., Kim, S., Yang, E., Hwang, S.J.: Scalable and order-robust continual learning with additive parameter decomposition. In: ICLR (2020)
Nguyen, C.V., Li, Y., Bui, T.D., Turner, R.E.: Variational continual learning. In: ICLR (2018)
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: ECCV (2018)
Lee, S.W., Kim, J.H., Jun, J., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. In: NeurIPS (2017)
Chaudhry, A., et al.: Continual learning with tiny episodic memories. ICML-W (2019)
Chaudhry, A., Gordo, A., Lopez-Paz, D., Dokania, P.K., Torr, P.: Using hindsight to anchor past knowledge in continual learning (2020)
Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. In: ICLR (2019)
Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. In: NeurIPS (2019)
Tulving, E.: Episodic memory: from mind to brain. Ann. Rev. Psychol. 53(1), 1–25 (2002)
Norman, K.A., O’Reilly, R.C.: Modeling hippocampal and neocortical contributions to recognition memory: a complementary-learning-systems approach. Psychol. Rev. 110(4), 611 (2003)
Ren, M., Iuzzolino, M.L., Mozer, M.C., Zemel, R.S.: Wandering within a world: online contextualized few-shot learning. arXiv preprint arXiv:2007.04546 (2020)
Ji, X., Henriques, J., Tuytelaars, T., Vedaldi, A.: Automatic recall machines: internal replay, continual learning and the brain. arXiv preprint arXiv:2006.12323 (2020)
Hsu, Y.C., Liu, Y.C., Kira, Z.: Re-evaluating continual learning scenarios: a categorization and case for strong baselines. In: NeurIPS-W (2018)
Riemer, M., et al.: Learning to learn without forgetting by maximizing transfer and minimizing interference. In: ICLR (2019)
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T.P., Wayne, G.: Experience replay for continual learning. In: NeurIPS (2019)
Loshchilov, I., Hutter, F.: Sgdr: stochastic gradient descent with warm restarts. In: ICLR (2017)
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: regularization strategy to train strong classifiers with localizable features. In: ICCV (2019)
Yin, H., et al.: Dreaming to distill: data-free knowledge transfer via deepinversion. In: CVPR (2020)
Zeno, C., Golan, I., Hoffer, E., Soudry, D.: Task agnostic continual learning using online variational bayes. arXiv preprint arXiv:1803.10123 (2018)
Hocquet, G., Bichler, O., Querlioz, D.: Ova-inn: continual learning with invertible neural networks. IJCNN (2020)
van de Ven, G.M., Tolias, A.S.: Generative replay with feedback connections as a general strategy for continual learning. arXiv preprint arXiv:1809.10635 (2018)
Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. ICML (2018)
Rannen, A., Aljundi, R., Blaschko, M.B., Tuytelaars, T.: Encoder based lifelong learning. In: CVPR (2017)
Mallya, A., Lazebnik, S.: Packnet: adding multiple tasks to a single network by iterative pruning. In: CVPR (2018)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: CVPR (2017)
Acknowledgements
AP would like to thank Aditya Bharti, Shyamgopal Karthik, Saujas Vaduguru, and Aurobindo Munagala for helpful discussions. PHS and PD thank EPSRC/MURI grant EP/N019474/1, and Facebook (DeepFakes grant) for their support. This project was supported by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme. PHS and PD also acknowledge FiveAI UK.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Prabhu, A., Torr, P.H.S., Dokania, P.K. (2020). GDumb: A Simple Approach that Questions Our Progress in Continual Learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12347. Springer, Cham. https://doi.org/10.1007/978-3-030-58536-5_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-58536-5_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58535-8
Online ISBN: 978-3-030-58536-5
eBook Packages: Computer ScienceComputer Science (R0)