Abstract
Reward mechanism of reinforcement learning alleviates the inconsistency between training and evaluation in neural machine translation. However, the model still incapable to learn ideal parameters when rewards are sparse or a weak sampling strategy is adopted. Therefore, we propose a reinforcement learning method based on prioritized experience replay to deal with the problems. The model experiences are obtained through reinforcement learning. Then they are stored in a experience buffer and assigned priorities according to the value of experience. The experience with higher priority in buffer will be extracted by model to optimize the parameters during training phase. To verify the robustness of our method, we not only conduct experiments on English-German and Chinese-English, but also perform on agglutinative language Mongolian-Chinese. Experimental results show that our work consistently outperforms the baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
http://allennlp.org/elmo/ ELMO, which fully consider contextual information has shown certain potential in semantic learning. It has strong modeling capabilities, meanwhile, the parameters and complexity are relatively small, which is convenient for model construction and training.
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1409.0473
Harris, C.M., Mandelbaum, J.: A note on convergence requirements for nonlinear maximum-likelihood estimation of parameters from mixture models. Comput. OR 12(2), 237–240 (1985). https://doi.org/10.1016/0305-0548(85)90048-6
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, Arizona, USA, 12–17 February 2016, pp. 2094–2100 (2016). http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12389
He, D., et al.: Dual learning for machine translation. In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, pp. 820–828 (2016). http://papers.nips.cc/paper/6469-dual-learning-for-machine-translation
Lin, L.J.: Self-improving reactive agents based on reinforcement learning, planning and teaching. Mach. Learn. 8, 293–321 (1992). https://doi.org/10.1007/BF00992699
Mnih, V., et al.: Playing Atari with deep reinforcement learning. CoRR abs/1312.5602 (2013). http://arxiv.org/abs/1312.5602
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 6–12 July 2002, pp. 311–318 (2002). https://www.aclweb.org/anthology/P02-1040/
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, 1–6 June 2018, (Volume 1: Long Papers), pp. 2227–2237 (2018). https://www.aclweb.org/anthology/N18-1202/
Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. In: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, 2–4 May 2016, Conference Track Proceedings (2016). http://arxiv.org/abs/1511.05952
Shen, S., et al.: Minimum risk training for neural machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 7–12 August 2016, Volume 1: Long Papers (2016). https://www.aclweb.org/anthology/P16-1159/
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, pp. 5998–6008 (2017). http://papers.nips.cc/paper/7181-attention-is-all-you-need
Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992). https://doi.org/10.1007/BF00992696
Wu, L., Tian, F., Qin, T., Lai, J., Liu, T.: A study of reinforcement learning for neural machine translation. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 31 October–4 November 2018, pp. 3612–3621 (2018). https://www.aclweb.org/anthology/D18-1397/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Sun, S., Hou, H., Wu, N., Guo, Z. (2020). Neural Machine Translation Based on Prioritized Experience Replay. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12397. Springer, Cham. https://doi.org/10.1007/978-3-030-61616-8_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-61616-8_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61615-1
Online ISBN: 978-3-030-61616-8
eBook Packages: Computer ScienceComputer Science (R0)