Abstract
This paper investigates a change of approach to textual data augmentation for sentiment classification, by switching from offline to online data modification. In other words, from changing the data before the training is started to using transformed samples during the training process. This allows utilizing the information about the current loss of the classifier. We try training with examples that maximize, minimize the loss, or are randomly sampled. We observe that the maximizing variant performs best in most cases. We use 2 neural network architectures, 3 data augmentation methods, and test them on 4 different datasets. Our experiments indicate that the switch to the online data augmentation improves the results for recurrent neural networks in all cases and for convolutional networks in some cases. The improvement reaches 2.3% above the baseline in terms of accuracy, averaged over all datasets, and 2.25% on one of the datasets, but averaged over dataset sizes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Harvard Kim CNN implementation. https://github.com/harvardnlp/sent-conv-torch
Sosuke Kobayashi’s data augmentation implementation. https://github.com/pfnet-research/contextual_augmentation
Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch. https://github.com/huggingface/transformers
WordNet online. https://wordnet.princeton.edu
Abulaish, M., Sah, A.K.: A text data augmentation approach for improving the performance of CNN. In: 2019 11th International Conference on Communication Systems & Networks (COMSNETS), pp. 625–630 (2019)
Anaby-Tavor, A., et al.: Do not have enough data? Deep learning to the rescue! In: AAAI, pp. 7383–7390 (2020)
Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Lechevallier Y., Saporta G. (eds.) Proceedings of COMPSTAT’2010, pp. 177–186. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-7908-2604-3_16
Coulombe, C.: Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint arXiv:1812.04718 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423
Fawzi, A., Samulowitz, H., Turaga, D., Frossard, P.: Adaptive data augmentation for image classification. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3688–3692. IEEE (2016)
Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 168–177. ACM (2004)
Jungiewicz, M., Smywiński-Pohl, A.: Towards textual data augmentation for neural networks: synonyms and maximum loss. Comput. Sci. 20(1) (2019). https://doi.org/10.7494/csci.2019.20.1.3023
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751. Association for Computational Linguistics (2014). https://doi.org/10.3115/v1/D14-1181
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)
Kobayashi, S.: Contextual augmentation: data augmentation by words with paradigmatic relations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 452–457. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/N18-2072
Ma, E.: nlpaug: data augmentation for NLP. https://github.com/makcedward/nlpaug. version 0.0.8 beta
Manning, C.D.: Computational linguistics and deep learning. Comput. Linguist. 41(4), 701–707 (2015)
Miller, G.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)
Miller, G.A.: Wordnet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)
Pang, B., Lee, L.: A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, p. 271. Association for Computational Linguistics (2004)
Quijas, J.K.: Analysing the effects of data augmentation and free parameters for text classification with recurrent convolutional neural networks. The University of Texas, El Paso (2017)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)
Tokui, S., Oono, K., Hido, S., Clayton, J.: Chainer: a next-generation open source framework for deep learning. In: Proceedings of Workshop on Machine Learning Systems (LearningSys) in the Twenty-Ninth Annual Conference on Neural Information Processing Systems (NIPS), vol. 5, pp. 1–6 (2015)
Wei, J., Zou, K.: EDA: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6383–6389. Association for Computational Linguistics, Hong Kong, China, November 2019
Wiebe, J., Wilson, T., Cardie, C.: Annotating expressions of opinions and emotions in language. Lang. Res. Eval. 39(2), 165–210 (2005). https://doi.org/10.1007/s10579-005-7880-9
Wu, X., Lv, S., Zang, L., Han, J., Hu, S.: Conditional BERT contextual augmentation. In: Rodrigues, J.M.F., et al. (eds.) ICCS 2019. LNCS, vol. 11539, pp. 84–95. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22747-0_7
Acknowledgment
This work was supported by the Polish National Centre for Research and Development – LIDER Program under Grant LIDER/27/0164/L-8/16/NCBR/2017 titled “Lemkin – intelligent legal information system”. We used the computational resources of the Prometheus computer of the PLGrid infrastructure for the experiments described in this paper.
We are grateful to Krzysztof Wróbel for his inspirational idea and to Agnieszka Jungiewicz for her remarks.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jungiewicz, M., Smywiński-Pohl, A. (2020). Data Augmentation for Sentiment Analysis in English – The Online Approach. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-61616-8_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61615-1
Online ISBN: 978-3-030-61616-8
eBook Packages: Computer ScienceComputer Science (R0)