Abstract
Deep reinforcement learning has obtained impressive performance in challenging tasks in recent years. Nevertheless, it has important limitations such as long training times and the number instances that are needed to achieve acceptable performance. Transfer learning offers an alternative to alleviate these limitations. In this paper, we propose a novel method for transferring knowledge from more than one source tasks. First, we select the best source tasks using a regressor that predicts the performance of a pre-trained model in the target task. Then, we apply a selection of relevant convolutional kernels for the target task in order to find a target model with similar number of parameters compared to the source ones. According to the results, our approach outperforms the accumulated reward obtained when learning from scratch in 20.62% using lower parameters (about 56% of the total, depending on the specific game).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bellemare, M.G., Naddaf, Y., Veness, J., Bowling, M.: The arcade learning environment: an evaluation platform for general agents. JAIR 47, 253–279 (2013)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Buitinck, L., et al.: API design for machine learning software: experiences from the scikit-learn project. In: ECML PKDD Workshop: Languages for Data Mining and Machine Learning, pp. 108–122 (2013)
Castro, P.S., Moitra, S., Gelada, C., Kumar, S., Bellemare, M.G.: Dopamine: a research framework for deep reinforcement learning (2018). http://arxiv.org/abs/1812.06110
de la Cruz, G., Du, Y., Irwin, J., Taylor, M.: Initial progress in transfer for deep reinforcement learning algorithms, July 2016
Cruz, Jr., G.V., Du, Y., Taylor, M.E.: Jointly pre-training with supervised, autoencoder, and value losses for deep reinforcement learning. arXiv preprint arXiv:1904.02206 (2019)
Hester, T., et al.: Deep Q-learning from demonstrations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Mittel, A., Munukutla, S., Yadav, H.: Visual transfer between Atari games using competitive reinforcement learning. arXiv preprint arXiv:1809.00397 (2018)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Pan, J., Wang, X., Cheng, Y., Yu, Q.: Multisource transfer double DQN based on actor learning. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2227–2238 (2018)
Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2009)
Parisotto, E., Ba, J.L., Salakhutdinov, R.: Actor-mimic: deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342 (2016)
Rusu, A.A., et al.: Policy distillation. arXiv preprint arXiv:1511.06295 (2015)
Schmitt, S., et al.: Kickstarting deep reinforcement learning. arXiv preprint arXiv:1803.03835 (2018)
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484 (2016)
Sutton, R.S., Barto, A.G., et al.: Introduction to Reinforcement Learning. MIT Press, Cambridge (2018)
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. JMLR 10(Jul), 1633–1685 (2009)
Yang, Q., Zhang, Y., Dai, W., Pan, S.J.: Transfer Learning. Cambridge University Press, Cambridge (2020)
Zhang, X., Ma, H.: Pretraining deep actor-critic reinforcement learning algorithms with expert demonstrations (2018)
Aknowledgements
The authors thankfully acknowledge computer resources, technical advice and support provided by Laboratorio Nacional de Supercómputo del Sureste de México (LNS), a member of CONACYT national laboratories with projects No. 201901047C and 202002030c. We also want to acknowledge the Laboratorio Nacional de Supercómputo del Bajio with project No. 2020.1. Jesús García-Ramírez acknowledges CONACYT for the scholarship that supports his PhD studies associated to CVU number 701191.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
García-Ramírez, J., Morales, E., Escalante, H.J. (2021). Multi-source Transfer Learning for Deep Reinforcement Learning. In: Roman-Rangel, E., Kuri-Morales, Á.F., Martínez-Trinidad, J.F., Carrasco-Ochoa, J.A., Olvera-López, J.A. (eds) Pattern Recognition. MCPR 2021. Lecture Notes in Computer Science(), vol 12725. Springer, Cham. https://doi.org/10.1007/978-3-030-77004-4_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-77004-4_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77003-7
Online ISBN: 978-3-030-77004-4
eBook Packages: Computer ScienceComputer Science (R0)