Abstract
Now a days, understanding the review of the articles, movies are the major issue due to different sentiment present on them. Reviews are short texts which expressing the opinion of the writer on certain texts and express the sentiment related to them. In the recent past, many researchers pay attention to sentiment analysis. In this research work, a novel binary sentiment classification is proposed to classify either positive or negative sentiment. First, the Bidirectional Encoder Representations from Transformers (BERT) embeddings are introduced to tokenize and preprocess the input text. The Bidirectional Long Short-Term Memory (BiLSTM) – Bidirectional Gated Recurrent Unit (BiGRU) and 1-D Convolutional Neural Network (CNN) model is integrated and proposed for sentiment classification. The proposed integrated BERT Embedding and BiLSTM-BiGRU is applied to extract the specified target and self-attention layer is added for better understanding of context, further 1-D CNN along with few other deep learning layers, the sentiment is classified for the selected IMDB movie review dataset. The proposed BERT Embedding + BiLSTM-BiGRU + self-attention and 1-D CNN model is trained and validated with the IMDB movie review dataset. From the simulation, it is found that the testing accuracy and AUC (Area Under the Curve) values are 93.89% and 0.9828 respectively. The performance of the proposed integrated BERT Embedding + BiLSTM-BiGRU+ self-attention and 1-D CNN model is compared with existing models and it is observed that it outperforms better in binary sentiment classification analysis.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of data and material
NA
Code availability
Custom Code.
References
Alaparthi S, Mishra M (2020) Bidirectional Encoder Representations from Transformers (BERT): a sentiment analysis Odyssey, arXiv:2007.01127, 01-15
Ali NM, Abd El Hamid MM, Youssif A (2019) Sentiment analysis for movies reviews dataset using deep learning models. Int J Data Min Knowl Manag Process 9(2/3):19–27. https://doi.org/10.5121/ijdkp.2019.9302
Amit M, Adi S (2016) Word Embeddings and their use in sentence classification tasks. arXiv:1610.082229v1
An H, Moon N (2019) Design of recommendation system for tourist spot using sentiment analysis based on CNN-LSTM. J Ambient Intell Humaniz Comput 13:1653–1663. https://doi.org/10.1007/s12652-019-01521-w
Appel O, Chiclana F, Carter J, Fujita H (2016) A hybrid approach to the sentiment analysis problem at the sentence level. Knowl-Based Syst 108:110–124
Bodapati JD, Veeranjaneyulu N, Shaik S (2019) Sentiment analysis from movie reviews using LSTMs. Ing Syst Inf 24(1):125–129. https://doi.org/10.18280/isi.240119
Bojanowski P, Grave E, Joulin A, Mikolov T (2017) Enriching word vectors with subword information. Trans Assoc Comput Linguist 5:135–146
Catal A, Nangir M (2017) A sentiment classification model based on multiple classifiers. Appl Soft Comput 50:135–141
Chaturvedi I, Ong Y, Tsang IW, Welsch RE, Cambria E (2016) Learning word dependencies in text by means of a deep recurrent belief network. Knowl-Based Syst 108:144–154
Chen Y, Zhang Z (2018) Research on text sentiment analysis based on CNNs and SVM, 2018 13th IEEE conference on industrial electronics and applications, pp 2731–2734
Cheng Y, Yao L, Xiang G, Zhang G, Tang T, Zhong L (2020) Text sentiment orientation analysis based on Multi-Channel CNN and bidirectional GRU with attention mechanism. IEEE Access 8:134964–134975. https://doi.org/10.1109/ACCESS.2020.3005823
Delvin J, Chang MW, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL- Human Language Technologies, pp 4171–4186
Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target dependent twitter sentiment classification. 52nd Annual Meeting of the Association for Computational Linguistics, 2, pp 49–54
Ganapathibhotla M, Liu B (2008) Mining opinions in comparative sentences. In: Proceedings of the 22nd international conference on computational linguistics, 1, pp 241–248
Garg K, Lobiyal DK (2021) KL-NF technique for sentiment classification. Multimed Tools Appl 80:19885–19907. https://doi.org/10.1007/s11042-021-10559-y
Gavilanes FM, López AT, Martínez JJ, Montenegro CE, Castaño GFJ (2016) Unsupervised method for sentiment analysis in online texts. Expert Syst Appl 58:57–75
Graves A, Mohamed AR, Hinton G (2013) Speech recognition with deep recurrent neural networks. In: IEEE international conference on acoustics, speech and signal processing, pp 6645–6649
Ivan T, Ryan MC (2008) Modeling online reviews with multi-grain topic models. Proceeding of the 17th international conference on world wide web 2008, pp 111–120, https://doi.org/10.1145/1367497.1367513
Jindal N, Liu B (2006) Identifying comparative sentences in text documents. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval, pp 244–251
Khan AH, Qamar U, Bashir S (2019) SentiMI: introducing point-wise mutual information with SentiWordNet to improve sentiment polarity detection. Appl Soft Comput 39:140–153
Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods. EMNLP, pp 1746–1751
Lai H, Yan X (2021) Multimodal sentiment analysis with asymmetric window multi-attentions. Multimed Tools Appl. https://doi.org/10.1007/s11042-021-11234-y
Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning, pp 1188–1196
Li D, Qian J (2016) Text sentiment analysis based on long short-term memory, first IEEE international conference on computer communication and the internet (ICCCI), pp 471–475. https://doi.org/10.1109/CCI.2016.7778967
Li W, Gao S, Zhou H, Huang Z, Zhang K, Li W (2019) The automatic text classification method based on BERT and feature union. 2019 IEEE 25th international conference on parallel and distributed systems, pp 774–777. https://doi.org/10.1109/ICPADS47876.2019.00114.
Liqiang N, Yi-Liang Z, Mohammad A, Jialie S, Tat-Seng C (2015) Bridging the vocabulary gap between health seekers and healthcare knowledge. IEEE Trans Knowl Data Eng 27:396–409
Liu B (2015) Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge University Press
Liu W, Cao G, Yin J (2019) Bi-level attention model for sentiment analysis of short texts. IEEE Access 7:13–22
Ma Y, Peng H, Cambria E (2018) Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://ojs.aaai.org/index.php/AAAI/article/view/12048
Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: a survey. Ain Shams Eng J 5(4):1093–1113
Minaee S, Azimi E, Abdolrashidi AA (2019) Deep-sentiment: sentiment analysis using ensemble of CNN and bi-LSTM models, arXiv:1904.04206v1
Mitchell M, Aguilar J, Wilson T, Durme BV (2013) Open domain targeted sentiment. In: Proceedings of the 2013 conference on empirical methods in natural language processing, pp 1643–1654
Munikar M, Shakya S, Shreshtha A (2019) Fine-grained sentiment classification using BERT. Artificial Intelligence for Transforming Business and Society, 01–05
Narayanan R, Liu B, Choudhary A (2009) Sentiment analysis of conditional sentences. In: Proceedings of the 2009 conference on empirical methods in natural language processing (EMNLP), vol 1, pp 180–189
Naz H, Ahuja S, Kumar D, Rishu (2021) DT-FNN based effective hybrid classification scheme for twitter sentiment analysis. Multimed Tools Appl 80:11443–11458. https://doi.org/10.1007/s11042-020-10190-3
Ortigos A, Martin JM, Carro RM (2014) Sentiment analysis in Facebook and its application to e-learning. Comput Hum Behav 31:527–541
Pennington J, Socher R, Manning C (2014) GloVe: global vectors for word representation. In: Proceedings of the 2014 conference on emperical methods in natural language processing, pp 1532–1543
Ravi K, Ravi V (2015) A survey on opinion mining and sentiment analysis: tasks, approaches and applications. Knowl-Based Syst 89:14–46
Rill S, Reinel D, Scheidt JZ, RV. (2014) PoliTwi: early detection of emerging political topics on twitter and the impact on concept level sentiment analysis. Knowl-Based Syst 69:24–33
Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117
Shaukat Z, Zulfiqar AA, Xiao C, Azeem M, Mahmood T (2020) Sentiment analysis on IMDB using lexicon and neural networks. SN Appl Sci 2:148. https://doi.org/10.1007/s42452-019-1926-x
Shen J, Liao X, Tao Z (2019) Sentence-level sentiment analysis via BERT and BiGRU. Proc. SPIE 11321, 2019 international conference on image and video processing, and artificial intelligence, 113212S. https://doi.org/10.1117/12.2550215
Singh NK, Tomar DS, Sangaiah AK (2020) Sentiment analysis: a review and comparative analysis over social media. J Ambient Intell Humaniz Comput 11:97–117
Tang D, Qin B, Liu T (2015) Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdiscip Rev: Data Min Knowl Discov 5(6):292–303. https://doi.org/10.1002/widm.1171
Tang D, Qin B, Wei F, Dong L, Liu T, Zhou M (2015) A joint segmentation and classification framework for sentence level sentiment classification. IEEE/ACM Trans Audio Speech Lang Process 23(11):1750–1761. https://doi.org/10.1109/TASLP.2015.2449071
Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. Proceedings of the 2016 conference on empirical methods in natural language processing, pp 214–224
Verma P, Khandelwal B (2019) Word embeddings and its application in deep learning. Int J Innov Technol Explor Eng 8(11):337–341
Victor M, Bracha S, Lior R (2016) .Language models with pre-trained (GloVe) word Embeddings. arXiv:1610.03759
Wadawadagi RS, Pagi VB (2019) An enterprise perspective of web content analysis research: a strategic road- map. Int J Knowl Web Intell 6(2):51–88
Wiebe J, Wilson T, Cardie C (2005) Annotating expressions of opinions and emotions in language. Lang Resour Eval 39(2–3):165–210
Xu K, Liao SS, Li J, Song Y (2011) Mining comparative opinions from customer reviews for competitive intelligence. Decis Support Syst 50(4):743–754
Yang S, Ko Y (2011) Extracting comparative entities and predicates from texts using comparative type classification. In: Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, 1, pp 1636–1644
Yu Q, Zhao H, Wang Z (2019) Attention-based bidirectional gated recurrent unit neural networks for sentiment analysis. In: Proceedings of the 2nd international conference on artificial intelligence and pattern recognition, pp 116–119. https://doi.org/10.1145/3357254.3357262
Zhang L, Wang S, Liu B (2017) Deep learning for sentiment analysis: a survey. Wires Data Min Knowl Discov 2018:E1253
Zhou K, Long F (2018) Sentiment analysis of text based on CNN and bi-directional LSTM model. Proceedings of the 24th international conference on automation and computing, pp 1–5. https://doi.org/10.23919/IConAC.2018.8749069
Funding
NA
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interest regarding the publication of this paper and that the work presented in this article is not supported by any funding agency.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Gupta, B., Prakasam, P. & Velmurugan, T. Integrated BERT embeddings, BiLSTM-BiGRU and 1-D CNN model for binary sentiment classification analysis of movie reviews. Multimed Tools Appl 81, 33067–33086 (2022). https://doi.org/10.1007/s11042-022-13155-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-13155-w