An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning
Pages 196 - 203
Abstract
Blood Glucose (BG) control, which involves maintaining individuals’ BG within a healthy range through extracorporeal insulin injections, is crucial for people with type 1 diabetes. Self-blood glucose control increases the risk of hypo/hyperglycemia. Individualized and automated BG control can be formulated as a reinforcement learning problem. In this paper, we transformed the BG control problem from a prolonged action effect-partially observable Markov decision process to a Markov decision process framework by applying an exponential decay model for drug concentration, considering drug action’s delayed and prolonged nature. We propose a novel multi-step deep reinforcement learning-based algorithm with a prioritized experience replay sampling named Multi-step DQN for BG (MDBG) to solve the problem. Compared with single-step bootstrapped updates, MDBG is more efficient and reduces the influence of biasing targets. It converges faster, achieves higher cumulative rewards than the benchmark, and improves the percentage of time the patient’s BG is within the target range. MDBG validates the effectiveness of multi-step deep reinforcement learning in BG control, helps explore the optimal glycemic control strategy tailored to individual patients, and is expected to be generalized to diverse patient profiles characterized by varying insulin sensitivities, lifestyles, and comorbidities to improve their survival.
References
[1]
Kanyin Liane Ong, Lauryn K Stafford, Susan A McLaughlin, Edward J Boyko, Stein Emil Vollset, Amanda E Smith, Bronte E Dalton, Joe Duprey, Jessica A Cruz, Hailey Hagins, et al. Global, regional, and national burden of diabetes from 1990 to 2021. The Lancet, 402(10397):203–234, 2023.
[2]
Miguel Tejedor, Ashenafi Zebene Woldaregay, and Fred Godtliebsen. Reinforcement learning application in diabetes blood glucose control: A systematic review. Artificial intelligence in medicine, 104(10397):101836, 2020.
[3]
Intenational Diabetes Federation. Idf diabetes atlas, tenth. International Diabetes, 2021.
[4]
Eleni Bekiari, Konstantinos Kitsios, Hood Thabit, Martin Tauschmann, Eleni Athanasiadou, Thomas Karagiannis, Anna-Bettina Haidich, Roman Hovorka, and Apostolos Tsapas. Artificial pancreas treatment for outpatients with type 1 diabetes: systematic review and meta-analysis. bmj, 361, 2018.
[5]
Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closed-loop blood glucose control. In Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens, editors, Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pages 508–536. PMLR, PMLR, 07–08 Aug 2020.
[6]
Satish K Garg, Stuart A Weinzimer, William V Tamborlane, Bruce A Buckingham, Bruce W Bode, Timothy S Bailey, Ronald L Brazg, Jacob Ilany, Robert H Slover, Stacey M Anderson, et al. Glucose outcomes with the in-home use of a hybrid closed-loop insulin delivery system in adolescents and adults with type 1 diabetes. Diabetes technology & therapeutics, 19(3):155–163, 2017.
[7]
Sohaib Mehmood, Imran Ahmad, Hadeeqa Arif, Umm E Ammara, and Abdul Majeed. Artificial pancreas control strategies used for type 1 diabetes control and treatment: a comprehensive analysis. Applied System Innovation, 3(3):31, 2020.
[8]
Melanie K Bothe, Luke Dickens, Katrin Reichel, Arn Tellmann, Björn Ellger, Martin Westphal, and Ahmed A Faisal. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert review of medical devices, 10(5):661–673, 2013.
[9]
Charlotte K Boughton and Roman Hovorka. New closed-loop insulin systems. Diabetologia, 64:1007–1015, 2021.
[10]
Miguel Tejedor, Sigurd Nordtveit Hjerde, Jonas Nordhaug Myhre, and Fred Godtliebsen. Evaluating deep q-learning algorithms for controlling blood glucose in in silico type 1 diabetes. Diagnostics, 13(19):3150, 2023.
[11]
Benjamin Ribba, Sherri Dudal, Thierry Lavé, and Richard W Peck. Model-informed artificial intelligence: reinforcement learning for precision dosing. Clinical Pharmacology & Therapeutics, 107(4):853–857, 2020.
[12]
Ian Fox and Jenna Wiens. Reinforcement learning for blood glucose control: Challenges and opportunities, 2019.
[13]
Jonas Nordhaug Myhre, Miguel Tejedor, Ilkka Kalervo Launonen, Anas El Fathi, and Fred Godtliebsen. In-silico evaluation of glucose regulation using policy gradient reinforcement learning for patients with type 1 diabetes mellitus. Applied Sciences, 10(18):6350, 2020.
[14]
Jinhao Zhu, Yinjia Zhang, Weixiong Rao, Qinpei Zhao, Jiangfeng Li, and Congrong Wang. Reinforcement learning for diabetes blood glucose control with meal information. In Bioinformatics Research and Applications: 17th International Symposium, ISBRA 2021, Shenzhen, China, November 26–28, 2021, Proceedings 17, pages 80–91. Springer, 2021.
[15]
Francesco Di Felice, Alessandro Borri, and Maria Domenica Di Benedetto. Deep reinforcement learning for closed-loop blood glucose control: two approaches. IFAC-PapersOnLine, 55(40):115–120, 2022.
[16]
Sumana Basu, Marc-André Legault, Adriana Romero-Soriano, and Doina Precup. On the challenges of using reinforcement learning in precision drug dosing: Delay and prolongedness of action effectss, 2023.
[17]
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017.
[18]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
[19]
Taiyu Zhu, Kezhi Li, Pau Herrero, and Pantelis Georgiou. Basal glucose control in type 1 diabetes using deep reinforcement learning: An in silico validation. IEEE Journal of Biomedical and Health Informatics, 25(4):1223–1232, 2020.
[20]
Chiara Dalla Man, Marc D Breton, and Claudio Cobelli. Physical activity into the meal glucose—insulin model of type 1 diabetes: In silico studies, 2009.
[21]
Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli. The uva/padova type 1 diabetes simulator: new features. Journal of diabetes science and technology, 8(1):26–34, 2014.
[22]
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[23]
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2016.
[24]
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR, 2016.
[25]
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988.
[26]
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2):229–235, 1999.
[27]
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of ICML’17, pages 449–458, Sydney, NSW, Australia, 2017. PMLR, JMLR.org.
[28]
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration, 2019.
[29]
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[30]
Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147–163. PMLR, 2017.
[31]
Daniel Lopez-Martinez, Patrick Eschenfeldt, Sassan Ostvar, Myles Ingram, Chin Hur, and Rosalind Picard. Deep reinforcement learning for optimal critical care pain management with morphine using dueling double-deep q networks. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 3960–3963. IEEE, 2019.
[32]
Harry Emerson, Matthew Guy, and Ryan McConville. Offline reinforcement learning for safer blood glucose control in people with type 1 diabetes. Journal of Biomedical Informatics, 142:104376, 2023.
[33]
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.
[34]
Xuanchen Xiang and Simon Foo. Recent advances in deep reinforcement learning applications for solving partially observable markov decision processes (pomdp) problems: Part 1—fundamentals and applications in games, robotics and natural language processing. Machine Learning and Knowledge Extraction, 3(3):554–581, 2021.
[35]
Lingheng Meng, Rob Gorbet, and Dana Kulić. Memory-based deep reinforcement learning for pomdps. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619–5626. IEEE, 2021.
[36]
J Fernando Hernandez-Garcia and Richard S Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target, 2019.
[37]
Wei Yuan, Yueyuan Li, Hanyang Zhuang, Chunxiang Wang, and Ming Yang. Prioritized experience replay-based deep q learning: Multiple-reward architecture for highway driving decision making. IEEE Robotics & Automation Magazine, 28(4):21–31, 2021.
[38]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937. PMLR, 2016.
[39]
Jingtao Qin, Nanpeng Yu, and Yuanqi Gao. Solving unit commitment problems with multi-step deep reinforcement learning. In 2021 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm), pages 140–145. IEEE, 2021.
[40]
Jinyu Xie. Simglucose v0.2.1, 2018.
[41]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Index Terms
- An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning
Recommendations
Reinforcement Learning for Diabetes Blood Glucose Control with Meal Information
Bioinformatics Research and ApplicationsAbstractThe blood glucose management of diabetics is essentially a control and optimization problem. The blood glucose level of patients is mainly influenced by diet and insulin dose. The goal of blood glucose management is to continuously control the ...
Closed‐loop blood glucose control for type 1 diabetes
AbstractType 1 diabetes (T1D) is a specific form of diabetes wherein the immune system destroys insulin‐producing β cells until complete inability to secrete insulin. Patients with T1D require insulin therapy to maintain blood glucose levels within the ...
A method to early detect blood glucose variations using continuous glucose monitoring system
SMO'09: Proceedings of the 9th WSEAS international conference on Simulation, modelling and optimizationThe modern research in the field of the treatment for type I diabetes mellitus include the artificial pancreas. In this paper the authors have used the mathematical analysis of the blood glucose time series recorded through continuous glucose monitoring ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
279 pages
ISBN:9798400717666
DOI:10.1145/3674658
Copyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- College of Information Science and Technology, Beijing University of Chemical Technology
- the Central Universities
Conference
ICBBT 2024
ICBBT 2024: 2024 16th International Conference on Bioinformatics and Biomedical Technology
May 24 - 26, 2024
Chongqing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 16Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)16
Reflects downloads up to 13 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format