Abstract
A surge in public opinions mining against various societal topics using publicly available off-the-shelf sentiment analysis tools is evident in recent times. Since sentiment analysis is a domain-dependent problem, and the majority of the tools are built for customer reviews, the suitability of using such existing off-the-the-shelf tools for a societal topic is subject to investigation. None of the existing studies has thoroughly investigated on societal issues. This paper systematically evaluates the performance of 10 popularly used off-the-shelf tools and 17 state-of-the-art machine learning techniques and investigates their strengths and weaknesses using various societal and non-societal topics.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
One of the state of USA
Obama-McCain Debate
Opinionated text in Twitter
Application Programming Interface
References
Abbasi, A., Hassan, A., & Dhar, M. (2014). Benchmarking twitter sentiment analysis tools. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC), 14, pp 26–31.
Akhtar, M.S., Kumar, A., Ekbal, A., & Bhattacharyya, P. (2016). A hybrid deep learning architecture for sentiment analysis. In Proceedings of the 26th International Conference on Computational Linguistics (COLING): Technical Papers, pp 482–493.
Al-Smadi, M., Qawasmeh, O., Al-Ayyoub, M., Jararweh, Y., & Gupta, B. (2017). Deep recurrent neural network vs. support vector machine for aspect-based sentiment analysis of arabic hotels’ reviews. Journal of Computational Science.
Baly, R., Hobeica, R., Hajj, H., El-Hajj, W., Shaban, K.B., & Al-Sallab, A. (2016). A meta-framework for modeling the human reading process in sentiment analysis. ACM Transactions on Information Systems (TOIS), 35(1), 1–21.
Burnap, P., Williams, M.L., Sloan, L., Rana, O., Housley, W., Edwards, A., Knight, V., Procter, R., & Voss, A. (2014). Tweeting the terror: modelling the social media reaction to the woolwich terrorist attack. Social Network Analysis and Mining, 4(1), 206.
Cao, X., MacNaughton, P., Deng, Z., Yin, J., Zhang, X., & Allen, J.G. (2018). Using twitter to better understand the spatiotemporal patterns of public sentiment: A case study in massachusetts, usa. International journal of environmental research and public health, 15(2), 250.
Catal, C., & Nangir, M. (2017). A sentiment classification model based on multiple classifiers. Applied Soft Computing, 50, 135–141.
Chen, S., Mao, J., Li, G., Ma, C., & Cao, Y. (2020). Uncovering sentiment and retweet patterns of disaster-related tweets from a spatiotemporal perspective–a case study of hurricane harvey. Telematics and Informatics, 47, 101326.
Church, K.W., & Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational linguistics, 16(1), 22–29.
Cui, H., Mittal, V., & Datar, M. (2006). Comparative experiments on sentiment classification for online product reviews. In Proceedings of the 21st National Conference on Artificial Intelligence, AAAI Press, 2, pp 1265–1270.
dos Santos, C., & Gatti, M. (2014). Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of the 25th International Conference on Computational Linguistics (COLING): Technical Papers, pp 69–78.
Garay, J., Yap, R., & Sabellano, M.J. (2019). An analysis on the insights of the anti-vaccine movement from social media posts using k-means clustering algorithm and vader sentiment analyzer. IOP Conference Series: Materials Science and Engineering, 482(1), 012043.
Giachanou, A., & Crestani, F. (2016). Like it or not: A survey of twitter sentiment analysis methods. ACM Computing Surveys (CSUR), 49(2), 28.
Gonçalves, P., Araújo, M., Benevenuto, F., & Cha, M. (2013). Comparing and combining sentiment analysis methods. In Proceedings of the first ACM conference on Online social networks, pp 27–38.
Huang, M., Qian, Q., & Zhu, X. (2017). Encoding syntactic knowledge in neural networks for sentiment classification. ACM Transactions on Information Systems (TOIS), 35(3), 1–27.
Jianqiang, Z., Xiaolin, G., & Xuejun, Z. (2018). Deep convolution neural networks for twitter sentiment analysis. IEEE Access, 6, 23253–23260.
Karamibekr, M., & Ghorbani, A.A. (2012). Sentiment analysis of social issues. In Proceedings of the International Conference on Social Informatics (SocialInformatics), pp 215–221.
Kouloumpis, E., Wilson, T., & Moore, J.D. (2011). Twitter sentiment analysis: The good the bad and the omg!. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), p 164.
Kušen, E., & Strembeck, M. (2018). Politics, sentiments, and misinformation: An analysis of the twitter discussion on the 2016 austrian presidential elections. Online Social Networks and Media, 5, 37–50.
Lerman, K., Arora, M., Gallegos, L., Kumaraguru, P., & Garcia, D. (2016). Emotions, demographics and sociability in twitter interactions. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), pp 201–210.
Lu, Y., Sakamoto, K., Shibuki, H., & Mori, T. (2017). Are deep learning methods better for twitter sentiment analysis?. In Proceedings of the 23rd Annual Meeting of Natural Language Processing (Japan), pp 787–790.
Maynard, D., & Bontcheva, K. (2016). Challenges of evaluating sentiment analysis tools on social media. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), pp 1142–1148.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in neural information processing systems, pp 3111–3119.
Mohammad, S.M., Sobhani, P., & Kiritchenko, S. (2017). Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT), 17(3), 26.
Mohammad, S.M., Zhu, X., Kiritchenko, S., & Martin, J. (2015). Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4), 480–499.
Mostafa, A.M. (2017). An evaluation of sentiment analysis and classification algorithms for arabic textual data. International Journal of Computer Applications, 158(3), 1–8.
Neppalli, V.K., Caragea, C., Squicciarini, A., Tapia, A., & Stehle, S. (2017). Sentiment analysis during hurricane sandy in emergency response. International Journal of Disaster Risk Reduction, 21, 213–222.
Nguyen, H., & Nguyen, M.-L. (2018). A deep neural architecture for sentence-level sentiment classification in twitter social networking. In Proceedings of the Computational Linguistics, pp 15–27.
On, J., Park, H.-A., & Song, T.-M. (2019). Sentiment analysis of social media on childhood vaccination: Development of an ontology. Journal of medical Internet research, 21(6), e13456.
Ouyang, X., Zhou, P., Li, CH, & Liu, L. (2015). Sentiment analysis using convolutional neural network. In Proceedings of the IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM), pp 2359–2364.
Öztürk, N., & Ayvaz, S. (2018). Sentiment analysis on twitter: A text mining approach to the syrian refugee crisis. Telematics and Informatics, 35(1), 136–147. https://doi.org/10.1016/j.tele.2017.10.006.
Pak, A., & Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In LREc, 10, pp 1320–1326.
Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing (EMNLP), 10, pp 79–86.
Ribeiro, F.N., Araújo, M., Gonçalves, P., Gonçalves, M.A., & Benevenuto, F. (2016). Sentibench-a benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1), 23.
Saif, H., He, Y., Fernandez, M., & Alani, H. (2016). Contextual semantics for sentiment analysis of twitter. Information Processing & Management, 52 (1), 5–19.
Severyn, A., & Moschitti, A. (2015). Twitter sentiment analysis with deep convolutional neural networks. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 959–962.
Shannon, C.E. (2001). A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1), 3–55.
Silva, N.F.F.D., Coletta, L.F.S., & Hruschka, E.R. (2016). A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Computing Surveys (CSUR), 49(1), 1–26.
Singh, P., Sawhney, R.S., & Kahlon, K.S. (2018). Sentiment analysis of demonetization of 500 & 1000 rupee banknotes by indian government. ICT Express, 4(3), 124–129. https://doi.org/10.1016/j.icte.2017.03.001.
Sobhani, P., Mohammad, S., & Kiritchenko, S. (2016). Detecting stance in tweets and analyzing its interaction with sentiment. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics, pp 159–169.
Tumasjan, A., Sprenger, T.O., Sandner, P.G., & Welpe, I.M. (2010). Predicting elections with twitter: What 140 characters reveal about political sentiment. Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), 10(1), 178–185.
Vargas, S., McCreadie, R., Macdonald, C., & Ounis, I. (2016). Comparing overall and targeted sentiments in social media during crises. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), pp 695–698.
Vilares, D., Alonso, M.A., & Gómez-Rodríguez, C. (2017). Supervised sentiment analysis in multilingual environments. Information Processing & Management, 53(3), 595–607.
Xia, R., Zong, C., & Li, S. (2011). Ensemble of feature sets and classification algorithms for sentiment classification. Information Sciences, 181(6), 1138–1152.
Zhou, G.-Y., & Huang, J.X. (2017). Modeling and mining domain shared knowledge for sentiment analysis. ACM Transactions on Information Systems (TOIS), 36(2), 1–36.
Acknowledgments
This work is partially funded by the Ministry of Electronics & Information Technology, Government of India.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Singh, L.G., Singh, S.R. Empirical study of sentiment analysis tools and techniques on societal topics. J Intell Inf Syst 56, 379–407 (2021). https://doi.org/10.1007/s10844-020-00616-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10844-020-00616-7