Abstract
Session-aware recommender systems capture user-specific preferences that emerge within multiple user sessions by leveraging the sequential nature of user interactions. Existing session-aware recommendation methods face challenges in finding the right balance between exploration and exploitation leading to less diverse recommendations and also suffering from overestimation bias. This bias problem refers to the tendency for value estimates to be higher than their true values resulting in slower convergence, suboptimal, and less diverse recommendations. This paper proposes a Double Deep Q-network based session-aware recommender system, DDQN-SaRS, which takes care of the overestimation bias and generates diverse recommendations capturing the user’s dynamic interests. The proposed system works in two phases. The first phase generates embedding for users, items, and sessions using a Graph Convolutional Network (GCN). The obtained embedding vectors are then given to Double Deep Q-Network (DDQN), a Double Deep Reinforcement Learning (DDRL) technique for suggesting items of interest to the user(s) in the second phase. DDQN decouples the task of action selection and action evaluation by utilizing two networks viz. main and target networks and resolves the overestimation bias problem while maintaining diversity in recommendations. The proposed system learns recommendation policies and corresponding rewards from a pure offline setting. It is validated on two real-world datasets: Diginetica from the CIKM Cup Challenge 2016 and Retailrocket from the Kaggle competition. Experimental results show that our proposed system, DDQN-SaRS outperformed various baseline algorithms viz. S-POP, Item-KNN, GRU4Rec, STAMP, HRNN, NSAR,IDSR, and GNN-GNF.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of data and materials
No datasets were generated. Datasets used in the current study are publicly available.
References
Aggarwal, C. C. (2016). Content-Based Recommender Systems. Recommender Systems: The Textbook (pp. 139–166). Cham: Springer International Publishing.
Bedi, P., Khurana, P., & Sharma, R. (2021). Session based recommendations using CNN-LSTM with fuzzy time series. In International conference on artificial intelligence and speech technology (pp. 432–446). https://doi.org/10.1007/978-3-030-95711-7_36
Cai, Q., Zhao, & Hu, H. (2018). Session-aware music recommendation via a generative model approach. Soft Computing, 22, 1023–1031. https://doi.org/10.1007/s00500-016-2407-4
Chen, H. (2021). A DQN-based Recommender system for item-list recommendation. In IEEE International conference on big data (big data) (pp. 5699–5702). https://doi.org/10.1109/BigData52589.2021.9671947
Chen, W., Ren, P., & Cai, F., et al. (2020). Improving end-to-end sequential recommendations with intent-aware diversication. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 175–184). https://doi.org/10.1145/3340531.3411897
Ding, Z., Huang, Y., & Yuan, H., et. al. (2020). Introduction to reinforcement learning. In Deep reinforcement learning: fundamentals, research and applications (pp. 47–123). Springer Singapore
Feng, L., Cai, Y., Wei, E., et al. (2022). Graph neural networks with global noise ltering for session-based recommendation. Neurocomputing, 472, 113–123. https://doi.org/10.1016/j.neucom.2021.11.068
Garg, D., Gupta, P., & Malhotra, P., et al. (2020). Batch-Constrained Distributional Reinforcement Learning for Session-based Recommendation. arXiv preprint, 1–15. https://doi.org/10.48550/arXiv.2012.08984
Girmay, M., Maglogiannis, V., Naudts, D., et al. (2021). Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based QLearning. Sensors, 21(21), 6977. https://doi.org/10.3390/s21216977
Hidasi, B., Karatzoglou, A., & Baltrunas, L., et al. (2015). Session-based Recommendations with Recurrent Neural Networks. arXiv preprint, 1–10. https://doi.org/10.48550/arXiv.1511.06939
Hu, L., Chen, Q., Zhao, H., et al. (2018). Neural Cross-Session Filtering: Next-Item Prediction Under Intra and Inter-Session Context. IEEE Intelligent Systems, 33(6), 57–67. https://doi.org/10.1109/MIS.2018.2881516
Jannach, D., Quadrana, M., & Cremonesi, P. (2022). Session-Based Recommender Systems. Recommender Systems Handbook (pp. 301–334). Springer, US: New York, NY.
Jiang, L., Huang, H., & Ding, Z. (2019). Path planning for intelligent robots based on deep Q-learning with experience replay and heuristic knowledge. IEEE/CAA Journal of Automatica Sinica, 7(4), 1179–1189. https://doi.org/10.1109/JAS.2019.1911732
Kumari, T., Sharma, R., & Bedi, P. (2022). A contextual-bandit approach for multifaceted reciprocal recommendations in online dating. Journal of Intelligent Information Systems, 59(3), 705–731. https://doi.org/10.1007/s10844-022-00708-6
Lei, Y., Pei, H., & Yan, H., et al. (2020). Reinforcement learning based recommendation with graph convolutional Q-network. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 1757–1760). https://doi.org/10.1145/3397271.3401237
Li, J., Ren, P., & Chen, Z., et al. (2017). Neural attentive session-based recommendation. In Proceedings of the 2017 ACM conference on information and knowledge management (pp .1419–1428). https://doi.org/10.1145/3132847.3132926
Linden, G., Smith, B., & York, J. (2003). Amazon.com recommendations: item-to-item collaborative ltering. IEEE Internet Computing, 7(1), 76–80. https://doi.org/10.1109/MIC.2003.1167344
Liu, C., Li, Y., Lin, H., et al. (2023). GNNRec: Gated graph neural network for session-based social recommendation model. Journal of Intelligent Information Systems, 60(1), 137–156. https://doi.org/10.1007/s10844-022-00733-5
Liu, Q., Zeng, Y., & Mokhosi, R., et al. (2018). STAMP: short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 1831–1839). https://doi.org/10.1145/3219819.3219950
Luo, X., Zhou, M., Xia, Y., et al. (2014). An Efficient Non-Negative Matrix- Factorization-Based Approach to Collaborative Filtering for Recommender Systems. IEEE Transactions on Industrial Informatics, 10(2), 1273–1284. https://doi.org/10.1109/TII.2014.2308433
Mollel, M. S., Abubakar, A. I., Ozturk, M., et al. (2020). Intelligent handover decision scheme using double deep reinforcement learning. Physical Communication, 42, 101–133. https://doi.org/10.1016/j.phycom.2020.101133
Muvunza, T., & Li, Y. (2023). Session-based recommendation with temporal dynamics for large volunteer networks. Journal of Intelligent Information Systems 1–22. https://doi.org/10.1007/s10844-023-00801-4
Nguyen, L. V., Hong, M. S., Jung, J. J., et al. (2020). Cognitive Similarity-Based Collaborative Filtering Recommendation System. Applied Science, 10(12), 4183. https://doi.org/10.3390/app10124183
Nguyen, L. V., Vo, Q. T., & Nguyen, T. H. (2023). Adaptive KNN-Based Extended Collaborative Filtering Recommendation Services. Big Data and Cognitive Computing, 7(2), 106. https://doi.org/10.3390/bdcc7020106
Pan, Z., Cai, F., & Chen, W., et al. (2020). Star graph neural networks for session-based recommendation. In Proceedings of the 29th ACM international conference on information & knowledge management (pp. 1195–1204). https://doi.org/10.1145/3340531.3412014
Pang, Y., Wu, L., Shen, Q., et al. (2022). Heterogeneous global graph neural networks for personalized session-based recommendation. In Proceedings of the fifteenth ACM international conference on web search and data mining (pp. 775–783). https://doi.org/10.1145/3488560.3498505
Phuong, T. M., Thanh, T. C., & Bach, N. X. (2019). Neural Session-Aware Recommendation. IEEE. Access, 7, 86884–86896. https://doi.org/10.1109/ACCESS.2019.2926074
Quadrana, M., Karatzoglou, A., Hidasi, B., et al. (2017). Personalizing session-based recommendations with hierarchical recurrent neural networks. In Proceedings of the eleventh ACM conference on recommender systems. (pp. 130–137). https://doi.org/10.1145/3109859.3109896
Rettig, L., Audiffren, J., & Cudre-Mauroux, P. (2019). Fusing vector space models for domain-specific applications. In IEEE 31st International conference on tools with artificial intelligence (ICTAI) (pp. 1110–1117). https://doi.org/10.1109/ICTAI.2019.00155
Richa, & Bedi, P. (2019). Parallel context-aware multi-agent tourism recommender system. International Journal of Computational Science and Engineering, 20(4), 536–549. https://doi.org/10.1504/IJCSE.2019.104440
Sewak, M. (2019). Deep Q network (DQN), double DQN, and dueling DQN: a step towards general artificial intelligence. In Deep reinforcement learning: frontiers of artificial intelligence (pp. 95–108). Springer Singapore
Sutton, R.S., & Barto, A.G. (2018). Reinforcement Learning: An Introduction. MIT Press
Tang, X., Chen, J., Pu, H., et al. (2021). Double Deep Reinforcement Learning- Based Energy Management for a Parallel Hybrid Electric Vehicle With Engine StartStop Strategy. IEEE Transactions on Transportation Electrification, 8(1), 1376–1388. https://doi.org/10.1109/TTE.2021.3101470
Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on artificial intelligence (pp. 2094–2100). https://doi.org/10.1609/aaai.v30i1.10295
Veličković, P. (2023). Everything is connected: Graph neural networks. Current Opinion in Structural Biology, 79, 102538. https://doi.org/10.1016/j.sbi.2023.102538
Wallach, D., & Goffinet, B. (1989). Mean squared error of prediction as a criterion for evaluating and comparing system models. Ecological Modelling, 44(3–4), 299–306. https://doi.org/10.1016/0304-3800(89)90035-5
Wu, S., Tang, Y., & Zhu, Y., et al. (2019). Session-based recommendation with graph neural networks. In Proceedings of the AAAI conference on artificial intelligence (pp. 346–353). https://doi.org/10.1609/aaai.v33i01.3301346
Xi, L., Yu, L., Xu, Y., et al. (2019). A Novel Multi-Agent DDQN-AD Method-Based Distributed Strategy for Automatic Generation Control of Integrated Energy Systems. IEEE Transactions on Sustainable Energy, 11(4), 2417–2426. https://doi.org/10.1109/TSTE.2019.2958361
Xia, X., Yin, H., Yu, J., et al. (2021). Self-supervised hypergraph convolutional networks for session-based recommendation. In Proceedings of the AAAI con-ference on artificial intelligence (pp. 4503–4511). https://doi.org/10.1609/aaai.v35i5.16578
Ying, H., Zhuang, F., & Zhang, F., et al. (2018). Sequential recommender system based on hierarchical attention network. In IJCAI international joint conference on artificial intelligence (pp. 3926–3932). https://doi.org/10.24963/ijcai.2018/546
Zhang, P., Guo, J., & Li, C., et al. (2023). Efficiently leveraging multi-level user intent for session-based recommendation via atten-mixer network. In Proceedings of the sixteenth ACM international conference on web search and data mining (pp. 168–176). https://doi.org/10.1145/3539597.3570445
Zhang, S., Tong, H., & Xu, J., et al. (2019). Graph convolutional networks: a comprehensive review. Computational Social Networks, 6(1), 1–23. https://doi.org/10.1186/s40649-019-0069-y
Zhao, J., Li, H., Qu, L., et al. (2022). DCFGAN: An adversarial deep reinforcement learning framework with improved negative sampling for session-based recommender systems. Information Sciences, 596, 222–235. https://doi.org/10.1016/j.ins.2022.02.04
Zhao, X., Gu, C., Zhang, H., et al. (2021). DEAR: deep reinforcement learning for online advertising impression in recommender systems. In Proceedings of the AAAI conference on artificial intelligence (750–758). https://doi.org/10.1609/aaai.v35i1.16156
Zhao, X., Xia, L., Zhang, L., et al. (2018). Deep reinforcement learning for page-wise recommendations. In Proceedings of the 12th ACM conference on recommender systems (pp. 95–103). https://doi.org/10.1145/3240323.3240374
Zhou, S., Dai, X., & Chen, H., et al. (2020). Interactive recommender system via knowledge graph-enhanced reinforcement learning. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval (pp. 179–188). https://doi.org/10.1145/3397271.3401174
Zou, L., Xia, L., & Ding, Z., et al. (2019). Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 2810–2818). https://doi.org/10.1145/3292500.3330668
Funding
No funding was received to assist with the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
Sr. Prof. Punam Bedi: Supervision, Reviewing, Editing, Resources Dr. Bhavna Gupta: Supervision, Writing, Review, Editing, Project Administration, Resources, Validation Dr. Ravish Sharma: Supervision, Writing, Review, Editing, Project Administration, Resources, Validation Ms. Purnima Khurana: Conceptualization, Methodology, Software, Resources, Validation, Investigation, Data curation, Formal analysis. Writing-Original draft preparation, Visualization, Writing- Reviewing and Editing.
Corresponding author
Ethics declarations
Ethics approval
‘Yes’
Consent to participate
‘Yes’
Consent for publication
‘Yes’
Conflict of Interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Khurana, P., Gupta, B., Sharma, R. et al. Session-aware recommender system using double deep reinforcement learning. J Intell Inf Syst 62, 403–429 (2024). https://doi.org/10.1007/s10844-023-00824-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10844-023-00824-x