Abstract
Recently, capturing task-specific and domain-specific patterns during pre-training has been shown to help models better adapt to downstream tasks. Existing methods usually use large-scale domain corpus and downstream supervised data to further pre-train pre-trained language models, which often brings a large computational burden and these data are difficult to obtain in most cases. To address these issues, we propose a pre-training method with a novel masking strategy called stepwise masking. The method employs stepwise masking to mine tokens related to the downstream task in mid-scale in-domain data and masks them. Then, the model is trained on these annotated data. In this stage, task-guided pre-training enables the model to learn task-specific and domain-specific patterns simultaneously and efficiently. Experimental results on sentiment analysis tasks show that our method can effectively improve the performance of the model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Araci, D.: FinBERT: financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063 (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Efroymson, M.A.: Multiple regression analysis. In: Mathematical Methods for Digital Computers, pp. 191–203 (1960)
Gu, Y., Zhang, Z., Wang, X., Liu, Z., Sun, M.: Train no evil: selective masking for task-guided pre-training. arXiv preprint arXiv:2004.09733 (2020)
Gururangan, S., et al.: Don’t stop pretraining: adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964 (2020)
He, R., McAuley, J.: Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering. In: Proceedings of the 25th International Conference on World Wide Web, pp. 507–517 (2016)
Huang, K., Altosaar, J., Ranganath, R.: ClinicalBERT: modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342 (2019)
Ke, P., Ji, H., Liu, S., Zhu, X., Huang, M.: SentiLARE: sentiment-aware language representation learning with linguistic knowledge. arXiv preprint arXiv:1911.02493 (2019)
Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
Liu, Z., Huang, D., Huang, K., Li, Z., Zhao, J.: FinBERT: a pre-trained financial language representation model for financial text mining. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 4513–4519 (2021)
Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., Manandhar, S.: SemEval-2014 task 4: aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 27–35. Association for Computational Linguistics, Dublin, Ireland, August 2014. https://doi.org/10.3115/v1/S14-2004, https://aclanthology.org/S14-2004
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding with unsupervised learning (2018)
Sun, C., Qiu, X., Xu, Y., Huang, X.: How to fine-tune BERT for text classification? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) CCL 2019. LNCS (LNAI), vol. 11856, pp. 194–206. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32381-3_16
Tian, H., et al.: SKEP: sentiment knowledge enhanced pre-training for sentiment analysis. arXiv preprint arXiv:2005.05635 (2020)
Wang, W., Pan, S.J., Dahlmeier, D., Xiao, X.: Recursive neural conditional random fields for aspect-based sentiment analysis. arXiv preprint arXiv:1603.06679 (2016)
Wu, H., Xu, K., Song, L., Jin, L., Zhang, H., Song, L.: Domain-adaptive pretraining methods for dialogue understanding. arXiv preprint arXiv:2105.13665 (2021)
Xu, S., et al.: K-plug: knowledge-injected pre-trained language model for natural language understanding and generation in e-commerce. arXiv preprint arXiv:2104.06960 (2021)
Yang, Z., et al.: XLNet: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Ye, Q., et al.: On the influence of masking policies in intermediate pre-training. arXiv preprint arXiv:2104.08840 (2021)
Zeng, J., Jiang, Y., Wu, S., Yin, Y., Li, M.: Task-guided disentangled tuning for pretrained language models. arXiv preprint arXiv:2203.11431 (2022)
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Zhu, Y., et al.: Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 19–27 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Pan, J., Ren, S., Rao, D., Zhao, Z., Xue, W. (2022). Stepwise Masking: A Masking Strategy Based on Stepwise Regression for Pre-training. In: Lu, W., Huang, S., Hong, Y., Zhou, X. (eds) Natural Language Processing and Chinese Computing. NLPCC 2022. Lecture Notes in Computer Science(), vol 13552. Springer, Cham. https://doi.org/10.1007/978-3-031-17189-5_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-17189-5_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-17188-8
Online ISBN: 978-3-031-17189-5
eBook Packages: Computer ScienceComputer Science (R0)