Abstract
The Transformer architecture has been widely used in sequence to sequence tasks since it was proposed. However, it only adds the representations of absolute positions to its inputs to make use of the order information of the sequence. It lacks explicit structures to exploit the reordering knowledge of words. In this paper, we propose a simple but effective method to incorporate the reordering knowledge into the Transformer translation system. The reordering knowledge of each word is obtained by an additional reordering-aware attention sublayer based on its semantic and contextual information. The proposed approach can be easily integrated into the existing framework of the Transformer. Experimental results on two public translation tasks demonstrate that our proposed method can achieve significant translation improvements over the basic Transformer model and also outperforms the existing competitive systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)
Brown, P.F., Pietra, S.D., Pietra, V.J.D., Mercer, R.L.: The mathematics of statistical machine translation: parameter estimation. Comput. Linguistics 19(2), 263–311 (1993)
Cettolo, M., Girardi, C., Federico, M.: Wit3: Web inventory of transcribed and translated talks. In: Conference of European Association for Machine Translation, pp. 261–268 (2012)
Chen, K., Wang, R., Utiyama, M., Sumita, E.: Neural machine translation with reordering embeddings. In: ACL (1), pp. 1787–1799. Association for Computational Linguistics (2019)
Chiang, D.: A hierarchical phrase-based model for statistical machine translation. In: ACL, pp. 263–270. The Association for Computer Linguistics (2005)
Gehring, J., Auli, M., Grangier, D., Dauphin, Y.N.: A convolutional encoder model for neural machine translation. In: ACL (1), pp. 123–135. Association for Computational Linguistics (2017)
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 1243–1252. PMLR (2017)
Kawara, Y., Chu, C., Arase, Y.: Recursive neural network based preordering for English-to-Japanese machine translation. In: ACL (3), pp. 21–27. Association for Computational Linguistics (2018)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (Poster) (2015)
Meng, F., Zhang, J.: DTMT: a novel deep transition architecture for neural machine translation. In: AAAI, pp. 224–231. AAAI Press (2019)
Nakagawa, T.: Efficient top-down BTG parsing for machine translation preordering. In: ACL (1), pp. 208–218. The Association for Computer Linguistics (2015)
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL (1). The Association for Computer Linguistics (2016)
Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: NAACL-HLT (2), pp. 464–468. Association for Computational Linguistics (2018)
So, D.R., Le, Q.V., Liang, C.: The evolved transformer. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5877–5886. PMLR (2019)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)
Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)
Zhang, J., Wang, M., Liu, Q., Zhou, J.: Incorporating word reordering knowledge into attention-based neural machine translation. In: ACL (1), pp. 1524–1534. Association for Computational Linguistics (2017)
Zhu, Z.: Evaluating neural machine translation in English-Japanese task. In: WAT, pp. 61–68. Workshop on Asian Translation (2015)
Acknowledgments
This research work has been funded by the National Natural Science Foundation of China (Grant No. 61772337), the National Key Research and Development Program of China NO. 2016QY03D0604.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, L., Zhou, J., Lu, W., Meng, K., Liu, G. (2020). Neural Machine Translation with Soft Reordering Knowledge. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Communications in Computer and Information Science, vol 1332. Springer, Cham. https://doi.org/10.1007/978-3-030-63820-7_79
Download citation
DOI: https://doi.org/10.1007/978-3-030-63820-7_79
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63819-1
Online ISBN: 978-3-030-63820-7
eBook Packages: Computer ScienceComputer Science (R0)