Abstract
Spoken Language Understanding (SLU) task is a wide application task in Natural Language Processing. In the success of the pre-trained BERT model, NLU is addressed by Intent Classification and Slot Filling task with significant improvement performance. However, classed imbalance problem in NLU has not been carefully investigated, while this problem in Semantic Parsing datasets is frequent. Therefore, this work focuses on diminishing this problem. We proposed a BERT-based architecture named JointBERT Classify Anonymous Entity (JointBERT-CAE) that improves the performance of the system on three Semantic Parsing datasets ATIS, Snips, ATIS Vietnamese, and a well-known Named Entity Recognize (NER) dataset CoNLL2003. In JointBERT-CAE architecture, we use multitask joint-learning to split conventional Slot Filling task into two sub-task, detect Anonymous Entity by Sequence tagging and Classify recognized anonymous entities tasks. The experimental results show the solid improvement of JointBERT-CAE when compared with BERT on all datasets, as well as the wide applicable capacity to other NLP tasks using the Sequence Tagging technique.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We use “word” to simplify, in practice, it should be sub-word split by BER Tokenizer.
- 2.
Downloaded from https://huggingface.co/bert-base-uncased.
- 3.
Downloaded from https://huggingface.co/vinai/phobert-base.
- 4.
Downloaded from https://huggingface.co/bert-base-cased.
- 5.
Downloaded from https://huggingface.co/roberta-large.
References
Castellucci, G., Bellomaria, V., Favalli, A., Romagnoli, R.: Multi-lingual intent detection and slot filling in a joint Bert-based model. CoRR, abs/1907.02884 (2019)
Chen, Q., Zhuo, Z., Wang, W.: Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909 (2019)
Coucke, A., et al.: Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190 (2018)
Dao, M.H., Truong, T.H., Nguyen, D.Q.: Intent detection and slot filling for Vietnamese. In: Proceedings of the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH) (2021)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, June 2019, pp. 4171–4186. Association for Computational Linguistics (2019)
Goo, C.-W., et al.: Slot-gated modeling for joint slot filling and intent prediction. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 753–757, New Orleans, Louisiana, June 2018. Association for Computational Linguistics (2018)
Grancharova, M., Berg, H., Dalianis, H.: Improving named entity recognition and classification in class imbalanced swedish electronic patient records through resampling. In: Eighth Swedish Language Technology Conference (SLTC). Förlag Göteborgs Universitet (2020)
Hardalov, M., Koychev, I., Nakov, P.: Enriched pre-trained transformers for joint slot filling and intent detection. CoRR, abs/2004.14848 (2020)
He, K., Yan, Y., Xu, W.: From context-aware to knowledge-aware: boosting OOV tokens recognition in slot tagging with background knowledge. Neurocomputing 445, 267–275 (2021)
Hemphill, C.T., Godfrey, J.J., Doddington, G.R.: The ATIS spoken language systems pilot corpus. In: Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, 24–27 June 1990 (1990)
Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: Proceedings of the Eighteenth International Conference on Machine Learning, ICML 2001, San Francisco, CA, USA, 2001, pp. 282–289. Morgan Kaufmann Publishers Inc (2001)
Li, C., Li, L., Qi, J.: A self-attentive model with gate mechanism for spoken language understanding. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October-November 2018, pp. 3824–3833. Association for Computational Linguistics (2018)
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., Li, J.: A unified MRC framework for named entity recognition. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5849–5859, Online, July 2020. Association for Computational Linguistics (2020)
Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., Li, J.: Dice loss for data-imbalanced NLP tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 465–476, Online, July 2020. Association for Computational Linguistics (2020)
Liu, Y., et al.: Roberta: a robustly optimized Bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Louvan, S., Magnini, B.: Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: a survey. In: Proceedings of the 28th International Conference on Computational Linguistics, pages 480–496, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics (2020)
Ma, X., Hovy, E.: End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064–1074, Berlin, Germany, August 2016. Association for Computational Linguistics (2016)
Nguyen, D.Q., Nguyen, A.T.: PhoBERT: pre-trained language models for Vietnamese. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1037–1042 (2020)
Qin, L., Che, W., Li, Y., Wen, H., Liu, T.: A stack-propagation framework with token-level intent detection for spoken language understanding. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2078–2087, Hong Kong, China, November 2019. Association for Computational Linguistics (2019)
Qin, L., Liu, T., Che, W., Kang, B., Zhao, S., Liu, T.: A co-interactive transformer for joint slot filling and intent detection. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8193–8197 (2021)
Qin, L., Xie, T., Che, W., Liu, T.: A survey on spoken language understanding: recent advances and new frontiers. In: Zhou, Z.-H. (ed.) Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4577–4584. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Survey Track (2021)
Qin, L., Xu, X., Che, W., Liu, T.: AGIF: an adaptive graph-interactive framework for joint multiple intent detection and slot filling. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1807–1816, Online, November 2020. Association for Computational Linguistics (2020)
Ravuri, S., Stolcke, A.: Recurrent neural network and LSTM models for lexical utterance classification. In: Proceedings Interspeech, pp. 135–139. ISCA - International Speech Communication Association, September 2015
Souza, F., Nogueira, R., de Alencar Lotufo, R.: Portuguese named entity recognition using BERT-CRF. arXiv:abs/1909.10649 (2019)
Tur, G., De Mori, R.: Spoken language understanding: systems for extracting semantic information from speech. In: Spoken Language Understanding: Systems for Extracting Semantic Information from Speech, March 2011
Vaswani, A., et al:. Attention is all you need. In: Guyon, I., et al. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates Inc (2017)
Wang, Y., Chu, H., Zhang, C., Gao, J.: Learning from language description: low-shot named entity recognition via decomposed framework. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1618–1630, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics (2021)
Wang, Y., Shen, Y., Jin, H.: A bi-model based RNN semantic frame parsing model for intent detection and slot filling. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 309–314, New Orleans, Louisiana, June 2018. Association for Computational Linguistics (2018)
Weld, H., Huang, X., Long, S., Poon, J., Han, S.C.: A survey of joint intent detection and slot-filling models in natural language understanding. arXiv preprint arXiv:2101.08091 (2021)
Xu, P., Sarikaya, R.: Convolutional neural network based triangular CRF for joint intent detection and slot filling. In: 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 78–83 (2013)
Acknowledgment
This work was supported by JSPS Kakenhi Grant Number 20H04295, 20K20406, and 20K20625.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Phuong, N.M., Le, T., Minh, N.L. (2022). CAE: Mechanism to Diminish the Class Imbalanced in SLU Slot Filling Task. In: Bădică, C., Treur, J., Benslimane, D., Hnatkowska, B., Krótkiewicz, M. (eds) Advances in Computational Collective Intelligence. ICCCI 2022. Communications in Computer and Information Science, vol 1653. Springer, Cham. https://doi.org/10.1007/978-3-031-16210-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-16210-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16209-1
Online ISBN: 978-3-031-16210-7
eBook Packages: Computer ScienceComputer Science (R0)