[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

An Enhanced Gated Recurrent Unit with Auto-Encoder for Solving Text Classification Problems

  • Research Article-Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

Classification has become an important task for automatically categorizing documents based on their respective group. The purpose of classification is to assign the pre-specified group or class to an instance based on the observed features related to that instance. For accurate text classification, feature selection techniques are normally used to identify important features and to remove irrelevant, undesired and noisy features for minimizing the dimensionality of feature space. Therefore, in this research, a new model namely Encoder Simplified GRU (ES-GRU) is proposed to reduce dimension of data using an auto-encoder (AE). Gated Recurrent Unit (GRU) is a deep learning algorithm that contains update gate and reset gate, which is considered as one of the most efficient text classification technique, specifically on sequential datasets. Accordingly, the reset gate is replaced with an update gate in order to reduce the redundancy and complexity in the standard GRU. The proposed model has been evaluated on five benchmark text datasets and compared with six baseline well-known text classification approaches, which includes standard GRU, AE, Long Short-Term Memory, Convolutional Neural Network, Support Vector Machine, and Naïve Bayes. Based on various types of performance evaluation parameters, a considerable amount of improvement has been observed in the performance of the proposed model as compared to state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Wang Z.; and Qu, Z. “Research on web text classification algorithm based on improved CNN and SVM,” IEEE, pp. 1958–1961, 2017.

  2. Sharif, W.; Samsudin, N.A; M. M. Deris, M.M and M. Aamir, “Improved relative discriminative criterion feature ranking technique for text classification.” Int. J. Artif. Intell., 15(2), pp. 61–78, 2017.

  3. Ahmed, R.; Al-Khatib, W.G. and Mahmoud, S.: “A Survey on handwritten documents word spotting.” Int. J. Multimed. Inf. Retr., 6(1), pp. 31–47, 2016 https://doi.org/10.1007/s13735-016-0110-y.

  4. Zulqarnain, M.; Ishak, S.A.; Ghazali, R.; Nawi, N.M.: An improved deep learning approach based on variant two-state gated recurrent unit and word embeddings for sentiment classification. Int. J. Adv. Comput. Sci. Appl. 11(1), 594–603 (2020)

    Google Scholar 

  5. Yi, J.; Zhang, Y.; Zhao, X. and Wan, J.: “A novel text clustering approach using deep-learning vocabulary network.” Math. Probl. Eng., 2017, 2017 https://doi.org/10.1155/2017/8310934.

  6. Barushka, A.; Hajek, P.: Spam filtering using integrated distribution-based balancing approach and regularized deep neural networks. Appl. Intell. 48(10), 3538–3556 (2018). https://doi.org/10.1007/s10489-018-1161-y

    Article  Google Scholar 

  7. Li, L.; Goh, T.T.; Jin, D.: How textual quality of online reviews affect classification performance: a case of deep learning sentiment analysis. Neural Comput. Appl. 32(9), 4387–4415 (2020). https://doi.org/10.1007/s00521-018-3865-7

    Article  Google Scholar 

  8. Jadhav, S.; Kasar, R.; Lade, N.; Patil, M.; Kolte, S.: Disease prediction by machine learning from healthcare communities. Int. J. Sci. Res. Sci. Technol. 5, 8869–8869 (2019). https://doi.org/10.32628/ijsrst19633

    Article  Google Scholar 

  9. Sharif, W.; Tun, U.; Onn, H.; Tri, I.; Yanto, R.: An optimised support vector machine with ringed seal search algorithm for efficient text classification. J. Eng. Sci. Technol. 14(3), 1601–1613 (2019)

    Google Scholar 

  10. Kowsari, K.; Brown, D.E.; Heidarysafa, M.; Meimandi, K.J.; Gerber, M.S. and Barnes, L.E.: “HDLTex: Hierarchical deep learning for text classification.” 2017 16th IEEE Int. Conf. Mach. Learn. Appl., pp. 364–371, 2017 https://doi.org/10.1109/ICMLA.2017.0-134.

  11. Dawar, M.: “Fast fuzzy feature clustering for text classification.” Acad. Ind. Res. Collab. Centre, Comput. Sci. Inf. Technol., 2, pp. 167–172, 2012 https://doi.org/10.5121/csit.2012.2317.

  12. Onan, A.; Korukoǧlu, S.; Bulut, H.: Ensemble of keyword extraction methods and classifiers in text classification. Expert Syst. Appl. 57, 232–247 (2016). https://doi.org/10.1016/j.eswa.2016.03.045

    Article  Google Scholar 

  13. Berger, M.J.: “Large scale multi-label text classification with semantic word vectors.” Tech. Rep., pp. 1–8, 2014.

  14. Yeh, C.K.; Wu, W.C.; Ko, W.J.; Wang, Y.C.F.: “Learning deep latent spaces for multi-label classification.” 31st AAAI Conf Artif. Intell. AAAI 2017, 2838–2844 (2017)

    Google Scholar 

  15. Xu, J.; Xu, C.; Zou, B.; Tang, Y.Y.; Peng, J. and You, X.: “New incremental learning algorithm with support vector machines.” IEEE Trans. Syst. Man, Cybern. Syst., vol. 49, no. 11, pp. 2230–2241, 2019 https://doi.org/10.1109/TSMC.2018.2791511.

  16. Xu, S.: Bayesian Naïve Bayes classifiers to text classification. J. Inf. Sci. 44(1), 48–59 (2018). https://doi.org/10.1177/0165551516677946

    Article  Google Scholar 

  17. Ghiassi, M.; Olschimke, M.; Moon, B.; Arnaudo, P.: Automated text classification using a dynamic artificial neural network model. Expert Syst. Appl. 39(12), 10967–10976 (2012). https://doi.org/10.1016/j.eswa.2012.03.027

    Article  Google Scholar 

  18. Liu, L.: Hierarchical learning for large multi-class network classification. Proc. Int. Conf. Pattern Recognit. (2016). https://doi.org/10.1109/ICPR.2016.7899980

    Article  Google Scholar 

  19. Hochreiter, S.: Long short term memory. Neural Comput. 9(8), 1–32 (1997). https://doi.org/10.1144/GSL.MEM.1999.018.01.02

    Article  MathSciNet  Google Scholar 

  20. Cho, K. et al. “Learning phrase representations using rnn encoder-decoder for statistical machine translation.” arXiv, pp. 1–15, 2014 https://doi.org/10.3115/v1/D14-1179.

  21. Zhou, G.: Minimal gated unit for recurrent neural networks. ICML 7, 153–163 (2016)

    Google Scholar 

  22. Kim, D.; Seo, D.; Cho, S.; Kang, P.: Multi-co-training for document classification using various document representations: TF–IDF, LDA, and Doc2Vec. Inf. Sci. (Ny) 477, 15–29 (2019). https://doi.org/10.1016/j.ins.2018.10.006

    Article  Google Scholar 

  23. Tang, D.; Qin, B. and Liu, T.: “Document modeling with gated recurrent neural network for sentiment classification.” Proc. 2015 Conf. Empir. Methods Nat. Lang. Process., pp. 1422–1432, 2015 https://doi.org/10.18653/v1/D15-1167.

  24. Conneau, A.; Schwenk, H.; Barrault, L. and Lecun, Y.: “Very deep convolutional networks for text classification,” arXiv, pp. 1–10, 2017.

  25. Kowsari, K.; Meimandi, K.J.; Heidarysafa, M.; Mendu, S.; Barnes, L.; Brown, D.: Text classification algorithms: A survey. Inf. 10(4), 1–68 (2019). https://doi.org/10.3390/info10040150

    Article  Google Scholar 

  26. Vincent, P.: “A Neural Probabilistic Language Model.” Neural Probabilistic Lang Model. J. Mach. Learn. Res. 3, 1137–1155 (2003)

    MATH  Google Scholar 

  27. Socher, R.; Huval, B.; Manning, C.D. and Ng, A.Y.: “Semantic compositionality through recursive matrix-vector spaces.” Proc. 2012 Jt. Conf. Empir. methods Nat. Lang. Process. Comput. Nat. Lang. Learn., pp. 1201–1211, 2012.

  28. Joulin, A.; Grave, E.; Bojanowski, P. and Mikolov, T.: “Bag of tricks for efficient text classification.” 2016 1511.09249v1.

  29. Prusa, J.D. and Khoshgoftaar, T.M.: “Designing a better data representation for deep neural networks and text classification,” Proc.-2016 IEEE 17th Int. Conf. Inf. Reuse Integr. IRI 2016, pp. 411–416, 2016 https://doi.org/10.1109/IRI.2016.61.

  30. Zhang, X.; Zhao, J. and Lecun, Y.: “Character-level convolutional networks for text classification.” Adv. Neural Inf. Process. Syst., vol. 2015 pp. 649–657, 2015.

  31. Chung, J.; Gulcehre, C.; Cho, K. and Bengio, Y.: “Empirical evaluation of gated recurrent neural networks on sequence modeling.” pp. 1–9, 2014 https://doi.org/10.1109/IJCNN.2015.7280624.

  32. Zhou, P.; Qi, Z.; Zheng, S.; Xu, J.; Bao, H. and Xu, B.: “Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling.” COLING 2016-26th Int. Conf. Comput. Linguist. Proc. COLING 2016 Tech. Pap., 2(1), pp. 3485–3495, 2016.

  33. Wei, J. and Zou, K.: “EDA: Easy data augmentation techniques for boosting performance on text classification tasks.” EMNLP-IJCNLP 2019-2019 Conf. Empir. Methods Nat. Lang. Process. 9th Int. Jt. Conf. Nat. Lang. Process. Proc. Conf., pp. 6382–6388, 2020 https://doi.org/10.18653/v1/d19-1670.

  34. Wang, Z. and Wu, Q.: “An Integrated Deep Generative Model for Text Classification and Generation,” Math. Probl. Eng., 2018, 2018 https://doi.org/10.1155/2018/7529286.

  35. Liao, R. et al.: “Reviving and improving recurrent back-propagation,” arXiv, pp. 3082–3091, 2018.

  36. Noaman, H.M.; Sarhan, S.S.; Rashwan, M.A.A.: Enhancing recurrent neural network-based language models by word tokenization. Human-centric Comput. Inf. Sci. 8(1), 1–13 (2018). https://doi.org/10.1186/s13673-018-0133-x

    Article  Google Scholar 

  37. Ghazali, R.; Husaini, N.A.; Ismail, L.H.; Herawan, T. and Hassim, Y.M.M.: “The performance of a Recurrent HONN for temperature time series prediction.” Proc. Int. Jt. Conf. Neural Networks, pp. 518–524, 2014 https://doi.org/10.1109/IJCNN.2014.6889789.

  38. Wang, Y.; Wang, H.; Zhang, X.; Chaspari, T.; Choe, Y. and Lu, M.: “An Attention-aware Bidirectional Multi-residual Recurrent Neural Network (Abmrnn): A Study about Better Short-term Text Classification,” ICASSP, IEEE Int. Conf. Acoust. Speech Signal Process. Proc., 2019, pp. 3582–3586, 2019 https://doi.org/10.1109/ICASSP.2019.8682565.

  39. Samarawickrama, A.J.P. and Fernando, T.G.I.: “A recurrent neural network approach in predicting daily stock prices an application to the Sri Lankan stock market.” 2017 IEEE Int. Conf. Ind. Inf. Syst. ICIIS 2017-Proc., 2018, pp. 1–6, 2018 https://doi.org/10.1109/ICIINFS.2017.8300345.

  40. Hochreiter, S.; Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  41. Sundermeyer, M.; Ney, H. and Schlüter, R.: “From feedforward to recurrent LSTM neural networks for language modeling.” IEEE/ACM Trans. Audio, Speech, Lang. Process, 23(3), pp. 517–529, 2015.

  42. Lee, H.: “For modeling sentences and documents.” Proc. 15th Annu. Conf. North Am. Chapter Assoc. Comput., pp. 1512–1521, 2015.

  43. Pascanu, R.; Tour, D.; Mikolov, T.; Tour, D.: On the difficulty of training recurrent neural networks. Conf. ICLR 2, 1310–1318 (2013)

    Google Scholar 

  44. Hao, Y.; Sheng, Y.; Wang, J.: Variant gated recurrent units with encoders to preprocess packets for payload-aware intrusion detection. IEEE Access 7, 49985–49998 (2019). https://doi.org/10.1109/ACCESS.2019.2910860

    Article  Google Scholar 

  45. Aamir, M.; Nawi, N.M.; Bin Mahdin, H.; Naseem, R. and Zulqarnain, M.: “Auto-encoder variants for solving handwritten digits classification problem.” Int. J. Fuzzy Log. Intell. Syst., 20(1), pp. 8–16, 2020 https://doi.org/10.5391/IJFIS.2020.20.1.8.

  46. Metwally, A.A.; Yu, P.S.; Reiman, D.; Dai, Y.; Finn, P.W.; Perkins, D.L.: Utilizing longitudinal microbiome taxonomic profiles to predict food allergy via long short-term memory networks. PLoS Comput. Biol. 15(2), 1–16 (2019).

    Article  Google Scholar 

  47. Pennington, J.; Socher, R. and Manning, C.D.: “GloVe : Global vectors for word representation.” Proc. Conf. Empir. Methods Nat. Lang. Process., pp. 1532–1543, 2014.

  48. Hinton, G.: Dropout: a simple way to prevent neural networks from overfitting. J. Machine Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  49. Chu, A.; Stratos, K. and Gimpel, K.: “Unsupervised label refinement improves dataless text classification,” arXiv, 2020.

  50. Shen, D. et al.: “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms,” ACL 2018-56th Annu. Meet. Assoc. Comput. Linguist. Proc. Conf. (Long Pap., 1, pp. 440–450, 2018 https://doi.org/10.18653/v1/p18-1041.

  51. Yogatama, D.; Dyer, C.; Ling, W.; Blunsom, P.: “Generative and discriminative text classification with recurrent neural networks.” arXiv, no. May, pp. 1–9, 2017.

  52. Kong, L.; Jiang, H.; Zhuang, Y.; Lyu, J.; Zhao, T.; and Zhang, C.: “ibrated language model fine-tuning for in- and out-of-distribution dataCal.” axXiv, pp. 1326–1340, 2020 https://doi.org/10.18653/v1/2020.emnlp-main.102.

  53. J. Xu and Q. Du, “A deep investigation into fasttext.” Proc. 21st IEEE Int. Conf. High Perform. Comput. Commun. 17th IEEE Int. Conf. Smart City 5th IEEE Int. Conf. Data Sci. Syst. HPCC/SmartCity/DSS 2019, pp. 1714–1719, 2019 https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00234.

  54. Wang, T.; Liu, L.; Zhang, H.; Zhang, L.; Chen, X.: Joint character-level convolutional and generative adversarial networks for text classification. Complexity 2020, 1–11 (2020). https://doi.org/10.1155/2020/8516216

    Article  Google Scholar 

  55. Ma, Y.; Fan, H.; Zhao, C.: Feature-based fusion adversarial recurrent neural networks for text sentiment classification. IEEE Access 7, 132542–132551 (2019). https://doi.org/10.1109/ACCESS.2019.2940506

    Article  Google Scholar 

  56. Fu, X.; Yang, J.; Li, J.; Fang, M.; Wang, H.: Lexicon-enhanced LSTM with attention for general sentiment analysis. IEEE Access 6, 71884–71891 (2018). https://doi.org/10.1109/ACCESS.2018.2878425

    Article  Google Scholar 

  57. Camacho-Collados, J. and Pilehvar, M. T.: “on the role of text preprocessing in neural network architectures: An evaluation study on text categorization and sentiment analysis,” arXiv Prepr. arXiv1707.01780, pp. 40–46, 2018, https://doi.org/10.18653/v1/w18-5406.

  58. Liu, B.: Text sentiment analysis based on CBOW model and deep learning in big data environment. J. Ambient Intell. Humaniz. Comput. 11(2), 451–458 (2020). https://doi.org/10.1007/s12652-018-1095-6

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Ministry of Education Malaysia, Universiti Tun Hussein Onn Malaysia and Research Management Center (RMC) for funding this research activity under the Fundamental Research Grant Scheme (FRGS), vote No.1641.

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, Universiti Tun Hussein Onn Malaysia, 86400, Parit Raja, Johor, Malaysia

    Muhammad Zulqarnain, Rozaida Ghazali, Yana Mazwin Mohmad Hassim & Muhammad Aamir

Authors
  1. Muhammad Zulqarnain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rozaida Ghazali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yana Mazwin Mohmad Hassim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Aamir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Zulqarnain.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zulqarnain, M., Ghazali, R., Hassim, Y.M.M. et al. An Enhanced Gated Recurrent Unit with Auto-Encoder for Solving Text Classification Problems. Arab J Sci Eng 46, 8953–8967 (2021). https://doi.org/10.1007/s13369-021-05691-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-021-05691-8

Keyword

Search

Navigation