Predominant Sense Acquisition with a Neural Random Walk Model
Pages 284 - 295
Abstract
Domain-Specific Senses (DSS) acquisition has been one of the major topics in Natural Language Processing (NLP). However, most results from unsupervised learning methods are not effective. This paper addresses the problem and proposes an approach for improving performance based on deep learning. To obtain DSS, we utilize Approximate Personalized Propagation of Neural Predictions (APPNP) consisting of Graph Convolutional Networks (GCN) and PageRank. GCN is a neural network that performs on graphs to learning sense features from neighbors’ senses and using Personalized PageRank for propagation. For constructing sense features, we collect glosses from WordNet and obtained sense embedding by using Bidirectional Encoder Representations from Transformers (BERT). Our experimental results show that the approach works well and attain at 0.614 Macro F1-score. In addition, to demonstrate the efficacy that DSS can work well in the NLP task, we apply the results on DSS to text categorization and gain a macro F1-score at 0.920, while the CNN baseline method is 0.776.
References
[1]
Koeling, R., McCarthy, D., Carroll, J.: Domain-specific sense distributions and predominant sense acquisition. In: Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 419–426 (2005)
[2]
Miller GA WordNet: a lexical database for English J. Commun. ACM 1995 38 39-41
[3]
Rose, T., Stevenson, M., Whitehead, M.: The Reuters corpus volume 1 - from yesterday’s news to tomorrow’s language resources. In: Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC 2002), pp. 29–31 (2002)
[4]
Devlin, J., Chang, M-W, Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. Journal of CoRR, arXiv:1810.04805, pp. 4171–4186 (2018)
[5]
Klicpera, J., Bojchevski, A., Günnemann, S.: Personalized embedding propagation: combining neural networks on graphs with personalized page rank. Journal of CoRR, arXiv:1810.05997 (2018)
[6]
Magnini, B., Cavaglià, G.: Integrating subject field codes into WordNet. In: Proceedings of the Second International Conference on Language and Evaluation (LREC 2000), pp. 1413–1418 (2000)
[7]
Gale, W.A., Church, K.W., Yarowsky, D.: One sense per discourse. In: Proceedings of the workshop and Speech and Natural Language, pp. 233–237 (1992)
[8]
Magnini, B., Strapparava, C., Pezzulo, G., Gliozzo, A.: Using domain information for word sense disambiguation. In: Proceedings of the the Second International Workshop on Evaluating Word Sense Disambiguation Systems, pp. 111–114 (2001)
[9]
Fukumoto, F., Suzuki, Y.: Identifying domain-specific senses and its application to text classification. In: Proceedings of the International Conference on Knowledge Engineering and Ontology Development, pp. 263–268 (2010)
[10]
Pasini T and Navigli R Train-o-Matic: supervised Word Sense Disambiguation with no (manual) effort J. Artif. Intell. 2019 279 103215
[11]
Mikolov, T., Chen K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of 1st International Conference on Learning Representations, ICLR 2013 (2013)
[12]
Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: Online learning of social representations. Journal of CoRR, arXiv:1403.6652 (2014)
[13]
Agirre E, Lacalle OL, and Soroa A Random walks for knowledge-based word sense disambiguation J. Comput. Linguist. 2014 40 57-84
[14]
Brin S and Page L The anatomy of a large-scale hypertextual web search engine J. Comput. Netw. 1998 30 107-117
[15]
Kipf, T. N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks. Journal of CoRR, arXiv:1609.02907 (2016)
[16]
Li, Q., Han, Z., Wu, X-M: Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning. Journal of CoRR, arXiv:1801.07606 (2018)
[17]
Kim, Y.: Convolutional Neural Networks for Sentence Classification. Journal of CoRR, arXiv:1408.5882 (2014)
[18]
Liu, J., Chang, W.C., Wu, Y., Yang, Y.: Deep learning for extreme multi-label text classification. In: Proceedings of 40th International ACM SIGIR conference on Research and Development in Information Retrieval (2017)
[19]
Wang, J., Wang, Z., Zhang, D., Yan, J.: Combining knowledge with deep convolutional neural networks for short text classification. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence IJCAI-17 (2017)
[20]
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014)
[21]
Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to document distances. In: Proceedings of Machine Learning Research, pp. 957–966 (2015)
[22]
Akiba, T., Sano, S, Yanase, T., Ohta, T., Koyama, M.: Optuna: a next-generation Hyperparameter Optimization Framework. Jounal of CoRR, arXiv:1907.10902 (2019)
[23]
Bevilacqua, M., Navigli, R.: Breaking through the 80% glass ceiling: raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2854–2864 (2020)
[24]
Yuan, D., Richardson, J., Doherty R. Evans, C., Altendorf, E.: Semi-supervised Word Sense Disambiguation with Neural Models. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 1374–1385 (2016)
[25]
Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Schol̈kopf, B.: Learning with local and global consistency. In: Proceedings of Advances in Neural Information Processing Systems, vol. 16, pp. 321–328 (2004)
Recommendations
Unsupervised translated word sense disambiguation in constructing bilingual lexical database
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingThe performance of a machine translation system depends on the availability of bilingual lexical dictionary and completion of its word sense disambiguation performance. Word sense disambiguation plays a vital role in several applications such as machine ...
Unsupervised WSD by finding the predominant sense using context as a dynamic thesaurus
We present and analyze an unsupervised method for Word Sense Disambiguation (WSD). Our work is based on the method presented by McCarthy et al. in 2004 for finding the predominant sense of each word in the entire corpus. Their maximization algorithm ...
Graph-based word sense disambiguation in Telugu language
In Natural Language Processing, word sense disambiguation (WSD) is an open challenge which improves the performance of the applications such as machine translation and information retrieval system. Many verbal languages will have many ambiguous words. ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Dec 2021
723 pages
ISBN:978-3-030-92237-5
DOI:10.1007/978-3-030-92238-2
© Springer Nature Switzerland AG 2021.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 08 December 2021
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Jan 2025