Abstract
Few-shot classification is a challenging task which aims to formulate the ability of humans to learn concepts from limited prior data. Recent progress in few-shot classification has featured meta-learning, in which a parameterized model for a learning algorithm is trained to learn the ability of handling classification tasks on extremely large episodes. In this work, we advance this few-shot classification paradigm by formulating it as a regular classification learning problem. We further propose multi-episode and cross-way training techniques, which respectively correspond to the minibatch and pretraining in regular classification problems for speeding up convergence in training. Experimental results on a state-of-the-art few-shot classification method (prototypical networks) demonstrate that both the proposed training strategies can highly accelerate the training process without accuracy loss for varying few-shot classification problems on Omniglot and miniImageNet.
Supported by the Beijing Nova Program of Science and Technology under Grant Z191100001119129 and the National Natural Science Foundation of China 61702520.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
It is noted that the network in pretraining does not need to be the same as that in the target supervised learning problem. Therefore, the cross-way training strategy can also work for optimization or memory-based methods where we can optionally just pretrain several feature layers. Related work can be explored in the future.
- 2.
- 3.
See results of cosine distance in supplementary material.
- 4.
See results of different learning rate scheduling strategy in supplementary material.
- 5.
Results are slightly different on miniImageNet because the open source code does not contain the implementations of training settings and data processing and we re-implement these by ourselves.
References
Lake, B., Salakhutdinov, R., Gross, J., Tenenbaum, J.: One shot learning of simple visual concepts. In: Proceedings of AMCSS, vol. 33, pp. 2568–2573 (2011)
Li, F.F., Rob, F., Pietro, P.: One-shot learning of object categories. IEEE Trans. PAMI 28(4), 594–611 (2006)
Schmidhuber, J.: Evolutionary principles in self-referential learning, On learning how to learn: The meta-meta-... hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich (1987)
Naik, D.K., Mammone, R.J.: Meta-neural networks that learn by learning. In: Proceedings of IJCNN, vol. 1, pp. 437–442. IEEE (1992)
Thrun, S., Pratt, L.: Learning to Learn. Springer, Heidelberg (2012). https://doi.org/10.1007/978-1-4615-5529-2
Hochreiter, S., Younger, A.S., Conwell, P.R.: Learning to learn using gradient descent. In: Dorffner, G., Bischof, H., Hornik, K. (eds.) ICANN 2001. LNCS, vol. 2130, pp. 87–94. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44668-0_13
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks, CoRR, vol. abs/1703.03400 (2017)
Ravi, S., Larochelle, H.: Optimization as a model for few-shot learning. In: Proceedings of ICLR (2017)
Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms, CoRR, vol. abs/1803.02999 (2018)
Rusu, A.A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., Hadsell, R.: Meta-learning with latent embedding optimization, arXiv preprint arXiv:1807.05960 (2018)
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., Lillicrap, T.: Meta-learning with memory-augmented neural networks. In: Proceedings of ICML, pp. 1842–1850 (2016)
Munkhdalai, T., Yu, H.: Meta networks. In: Proceedings of ICML, pp. 2554–2563. JMLR. org (2017)
Mishra, N., Rohaninejad, M., Chen, X., Abbeel, P.: Meta-learning with temporal convolutions, CoRR, vol. abs/1707.03141 (2017)
Koch, G., Zemel, R., Salakhutdinov, R.: Siamese neural networks for one-shot image recognition. In: ICML Deep Learning Workshop, vol. 2 (2015)
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: NIPS, pp. 3630–3638 (2016)
Shyam, P., Gupta, S., Dukkipati, A.: Attentive recurrent comparators. In: Proceedings of ICML, pp. 3173–3181. JMLR. org (2017)
Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning. In: NIPS, pp. 4077–4087 (2017)
Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: relation network for few-shot learning. In: Proceedings of IEEE CVPR, pp. 1199–1208 (2018)
Ren, M., et al.: Meta-learning for semi-supervised few-shot classification, CoRR, vol. abs/1803.00676 (2018)
Zhang, C., Cai, Y., Lin, G., Shen, C.: DeepEMD: few-shot image classification with differentiable earth mover’s distance and structured classifiers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12203–12213 (2020)
Chen, Y., Wang, X., Liu, Z., Xu, H., Darrell, T.: A new meta-baseline for few-shot learning, arXiv preprint arXiv:2003.04390 (2020)
Li, A., Huang, W., Lan, X., Feng, J., Li, Z., Wang, L.: Boosting few-shot learning with adaptive margin loss. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12576–12584 (2020)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. IJCV 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22(3), 400–407 (1951)
Dekel, O., Ran, G.B., Shamir, O., Xiao, L.: Optimal distributed online prediction using mini-batches. JMLR 13(1), 165–202 (2012)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99 (2015)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of IEEE ICCV, pp. 2961–2969 (2017)
He, K., Girshick, R., Dollár, P.: Rethinking ImageNet pre-training, arXiv preprint arXiv:1811.08883 (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014)
Yao, Z., Gholami, A., Lei, Q., Keutzer, K., Mahoney, M.W.: Hessian-based analysis of large batch training and robustness to adversaries. In: NeurIPS (2018)
Lee, J.D., Panageas, I., Piliouras, G., Simchowitz, M., Jordan, M.I., Recht, B.: First-order methods almost always avoid saddle points, ArXiv, vol. abs/1710.07406 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, M., Xu, Y., Bao, W., Xiang, X. (2021). Training Few-Shot Classification via the Perspective of Minibatch and Pretraining. In: Fang, L., Chen, Y., Zhai, G., Wang, J., Wang, R., Dong, W. (eds) Artificial Intelligence. CICAI 2021. Lecture Notes in Computer Science(), vol 13069. Springer, Cham. https://doi.org/10.1007/978-3-030-93046-2_55
Download citation
DOI: https://doi.org/10.1007/978-3-030-93046-2_55
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93045-5
Online ISBN: 978-3-030-93046-2
eBook Packages: Computer ScienceComputer Science (R0)