More Web Proxy on the site http://driver.im/

research-article

Public Access

Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatment

Authors:

Stephen L. Schmidt,

Afsana Chowdhury,

Jennifer J. Peters,

Katherine Genty,

Warren M. Grill,

Dennis A. Turner,

Miroslav PajicAuthors Info & Claims

ICCPS '23: Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023)

Pages 44 - 55

https://doi.org/10.1145/3576841.3585925

Published: 09 May 2023 Publication History

Abstract

Deep brain stimulation (DBS) has shown great promise toward treating motor symptoms caused by Parkinson's disease (PD), by delivering electrical pulses to the Basal Ganglia (BG) region of the brain. However, DBS devices approved by the U.S. Food and Drug Administration (FDA) can only deliver continuous DBS (cDBS) at a fixed amplitude; this energy inefficient operation reduces battery lifetime of the device, cannot adapt treatment dynamically for activity, and may cause significant side-effects (e.g., gait impairment). In this work, we introduce an offline reinforcement learning (RL) framework, allowing the use of past clinical data to train an RL policy to adjust the stimulation amplitude in real time, with the goal of reducing energy use while maintaining the same level of treatment (i.e., control) efficacy as cDBS. Moreover, clinical protocols require the safety and performance of such RL controllers to be demonstrated ahead of deployments in patients. Thus, we also introduce an offline policy evaluation (OPE) method to estimate the performance of RL policies using historical data, before deploying them on patients. We evaluated our framework on four PD patients equipped with the RC+S DBS system, employing the RL controllers during monthly clinical visits, with the overall control efficacy evaluated by severity of symptoms (i.e., bradykinesia and tremor), changes in PD biomakers (i.e., local field potentials), and patient ratings. The results from clinical experiments show that our RL-based controller maintains the same level of control efficacy as cDBS, but with significantly reduced stimulation energy. Further, the OPE method is shown effective in accurately estimating and ranking the expected returns of RL controllers.

References

[1]

Mattia Arlotti, Manuela Rosa, et al. 2016. The adaptive deep brain stimulation challenge. Parkinsonism & related disorders 28 (2016), 12--17.

[2]

Mattia Arlotti, Lorenzo Rossi, et al. 2016. An external portable device for adaptive deep brain stimulation (aDBS) clinical research in advanced Parkinson's Disease. Medical engineering & physics 38, 5 (2016), 498--505.

[3]

Alim Louis Benabid. 2003. Deep brain stimulation for Parkinson's disease. Current opinion in neurobiology 13, 6 (2003), 696--706.

[4]

Aleksandar Beric, Patrick J Kelly, et al. 2001. Complications of deep brain stimulation surgery. Stereotactic and functional neurosurgery 77, 1--4 (2001), 73--78.

[5]

M Beudel and P Brown. 2016. Adaptive deep brain stimulation in Parkinson's disease. Parkinsonism & related disorders 22 (2016), S123--S126.

[6]

Christopher Bishop. 2006. Pattern recognition and machine learning. Springer.

Digital Library

[7]

Peter Brown, Antonio Oliviero, et al. 2001. Dopamine dependency of oscillations between subthalamic nucleus and pallidum in Parkinson's disease. Journal of Neuroscience 21, 3 (2001), 1033--1038.

[8]

A H Butt, E Rovini, et al. 2018. Objective and automatic classification of Parkinson disease with Leap Motion controller. Biomedical engineering 17, 1 (2018), 1--21.

[9]

Witney Chen, Lowry Kirkby, et al. 2021. The role of large-scale data infrastructure in developing next-generation deep brain stimulation therapies. Frontiers in Human Neuroscience 15 (2021), 717401.

[10]

Bo Dai, Ofir Nachum, et al. 2020. Coindice: Off-policy confidence interval estimation. arXiv preprint arXiv:2010.11652 (2020).

[11]

Lonneke ML De Lau and Monique MB Breteler. 2006. Epidemiology of Parkinson's disease. The Lancet Neurology 5, 6 (2006), 525--535.

[12]

Günther Deuschl, Carmen Schade-Brittinger, et al. 2006. A randomized trial of deep-brain stimulation for Parkinson's disease. New England Journal of Medicine 355, 9 (2006), 896--908.

[13]

Kenneth A Follett, Frances M Weaver, et al. 2010. Pallidal versus subthalamic deep-brain stimulation for Parkinson's disease. New England Journal of Medicine 362, 22 (2010), 2077--2091.

[14]

Justin Fu, Mohammad Norouzi, et al. 2020. Benchmarks for Deep Off-Policy Evaluation. In ICLR.

[15]

Ge Gao, Qitong Gao, et al. 2022. A Reinforcement Learning-Informed Pattern Mining Framework for Multivariate Time Series Classification. In IJCAI.

[16]

Ge Gao, Song Ju, Markel Sanz Ausin, and Min Chi. 2023. Hope: Human-centric off-policy evaluation for e-learning and healthcare. In AAMAS.

[17]

Qitong Gao, Ge Gao, Min Chi, and Miroslav Pajic. 2023. Variational Latent Branching Model for Off-Policy Evaluation. In ICLR.

[18]

Qitong Gao, Davood Hajinezhad, et al. 2019. Reduced Variance Deep Reinforcement Learning with Temporal Logic Specifications. In ICCPS. ACM.

[19]

Qitong Gao, Michael Naumann, et al. 2020. Model-Based Design of Closed Loop Deep Brain Stimulation Controller using Reinforcement Learning. In 2020 ACM/IEEE 11th Int. Conf. on Cyber-Physical Systems (ICCPS). IEEE, 108--118.

[20]

Qitong Gao, Stephen L Schmidt, et al. 2022. Offline Policy Evaluation for Learning-based Deep Brain Stimulation Controllers. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS). IEEE, 80--91.

[21]

Qitong Gao, Dong Wang, et al. 2022. Gradient Importance Learning for Incomplete Observations. In International Conference on Learning Representations.

[22]

Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. 2017. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In Int. Conf. on robotics and automation (ICRA). IEEE, 3389--3396.

Digital Library

[23]

A. Guez, R. D. Vincent, M. Avoli, and J. Pineau. 2008. Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning. In AAAI. 1671--1678.

[24]

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML. PMLR, 1861--1870.

[25]

J. Habets, M. Heijmans, et al. 2018. An update on adaptive deep brain stimulation in Parkinson's disease. Movement Disorders 33, 12 (2018), 1834--1843.

[26]

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, and others. [n.d.]. Distilling the knowledge in a neural network. ([n. d.]).

[27]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.

Digital Library

[28]

S. Ishii, W. Yoshida, and J. Yoshimoto. 2002. Control of exploitation-exploration meta-parameter in reinforcement learning. Neural networks 15 (2002), 665--687.

[29]

Nan Jiang and Lihong Li. 2016. Doubly robust off-policy value evaluation for reinforcement learning. In ICML. PMLR, 652--661.

[30]

Ilija Jovanov, Michael Naumann, et al. 2018. Platform for model-based design and testing for deep brain stimulation. In ICCPS.

[31]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[32]

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).

[33]

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. 2022. Offline Reinforcement Learning with Implicit Q-Learning. In ICLR.

[34]

A.A. Kühn, A. Kupsch, GH. Schneider, and P Brown. 2006. Reduction in subthalamic 8--35 Hz oscillatory activity correlates with clinical improvement in Parkinson's disease. Euro. J. of Neuroscience 23, 7 (2006), 1956--1960.

[35]

Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics 22, 1 (1951), 79--86.

[36]

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative q-learning for offline reinforcement learning. In NeurIPS.

[37]

Alexis M Kuncel and Warren M Grill. 2004. Selection of stimulus parameters for deep brain stimulation. Clinical neurophysiology 115, 11 (2004), 2431--2441.

[38]

Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. 2020. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems 33 (2020), 741--752.

[39]

Timothy P Lillicrap, Jonathan J Hunt, et al. 2016. Continuous control with deep reinforcement learning. ICLR (2016).

[40]

Simon Little, Alex Pogosyan, et al. 2013. Adaptive deep brain stimulation in advanced Parkinson disease. Annals of neurology 74, 3 (2013), 449--457.

[41]

Simon Little, Elina Tripoliti, et al. 2016. Adaptive deep brain stimulation for Parkinson's disease demonstrates reduced speech side effects compared to conventional stimulation in the acute setting. J Neurol Neurosurg Psychiatry 87, 12 (2016), 1388--1389.

[42]

Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. 2018. Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation. In NeurIPS.

[43]

Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60.

[44]

C Marras, JC Beck, et al. 2018. Prevalence of Parkinson's disease across North America. NPJ Parkinson's disease 4, 1 (2018), 21.

[45]

Volodymyr Mnih, Adria Puigdomenech Badia, et al. 2016. Asynchronous methods for deep reinforcement learning. In ICML. 1928--1937.

[46]

Volodymyr Mnih, Koray Kavukcuoglu, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.

[47]

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. 2019. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. NeurIPS 32 (2019).

[48]

Vivek Nagaraj, Andrew Lamperski, and Theoden I Netoff. 2017. Seizure control in a computational model using a reinforcement learning stimulation paradigm. International J. of Neural Sys. 27, 07 (2017), 1750012.

[49]

Michael S Okun. 2012. Deep-brain stimulation for Parkinson's disease. New England Journal of Medicine 367, 16 (2012), 1529--1538.

[50]

Enrico Opri, Stephanie Cernera, et al. 2020. Chronic embedded cortico-thalamic closed-loop deep brain stimulation for the treatment of essential tremor. Science translational medicine 12, 572 (2020), eaay7680.

[51]

Bahram Parvinian, Christopher Scully, et al. 2018. Regulatory considerations for physiological closed-loop controlled medical devices used for automated critical care: food and drug administration workshop discussion topics. Anesthesia and analgesia 126, 6 (2018), 1916.

[52]

J. Pineau, A. Guez, et al. 2009. Treating epilepsy via adaptive neurostimulation: a reinforcement learning approach. Int. J. of Neural Sys. 19, 04 (2009), 227--240.

[53]

Rob Powers, Maryam Etezadi-Amoli, et al. 2021. Smartwatch inertial sensors continuously monitor real-world motor fluctuations in Parkinson's disease. Science translational medicine 13, 579 (2021), eabd7865.

[54]

Doina Precup. 2000. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series (2000), 80.

Digital Library

[55]

Claudia Ramaker, Johan Marinus, Anne Margarethe Stiggelbout, and Bob Johannes Van Hilten. 2002. Systematic evaluation of rating scales for impairment and disability in Parkinson's disease. Movement disorders 17, 5 (2002), 867--876.

[56]

Andrei A Rusu, Sergio G Colmenarejo, et al. 2016. Policy Distillation. In ICLR.

[57]

David Silver, Guy Lever, et al. 2014. Deterministic policy gradient algorithms.

[58]

Rosa Q So, Alexander R Kent, and Warren M Grill. 2012. Relative contributions of local cell and passing fiber activation and silencing to changes in thalamic fidelity during deep brain stimulation and lesioning: a computational modeling study. Journal of computational neuroscience 32, 3 (2012), 499--519.

Digital Library

[59]

Scott Stanslaski, Jeffrey Herron, et al. 2018. A chronically implantable neural coprocessor for investigating the treatment of neurological disorders. IEEE transactions on biomedical circuits and systems 12, 6 (2018), 1230--1245.

[60]

Nicole C Swann, Coralie de Hemptinne, et al. 2016. Gamma oscillations in the hyperkinetic state detected with chronic human brain recordings in Parkinson's disease. Journal of Neuroscience 36, 24 (2016), 6445--6458.

[61]

Ziyang Tang, Yihao Feng, et al. 2019. Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation. In ICLR.

[62]

Philip Thomas and Emma Brunskill. 2016. Data-efficient off-policy policy evaluation for reinforcement learning. In ICML. PMLR, 2139--2148.

[63]

Joshua K Wong, Günther Deuschl, et al. 2022. Proc. the 9th Annual Deep Brain Stimulation Think Tank: Advances in Cutting Edge Technologies, Artificial Intelligence, Neuromodulation, Neuroethics, Pain, Interventional Psychiatry, Epilepsy, and Traumatic Brain Injury. Frontiers in Human Neuroscience (2022), 25.

[64]

Yuhuai Wu, Elman Mansimov, et al. 2017. Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In NeurIPS.

[65]

Mengjiao Yang, Ofir Nachum, et al. 2020. Off-Policy Evaluation via the Regularized Lagrangian. In NeurIPS, Vol. 33.

Cited By

Pan MSchrum MMyers VBiyik EDragan ASalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Coprocessor actor criticProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693662(39292-39307)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693662
Gao GYang XChi MWooldridge MDy JNatarajan S(2024)Get a head startProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i11.29102(12136-12144)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i11.29102
Mehregan JYuan XBagwe GJackson JMohammed EZhang LYu C(2024)Enhancing Adaptive Deep Brain Stimulation via Efficient Reinforcement Learning2024 IEEE Intelligent Mobile Computing (MobileCloud)10.1109/MobileCloud62079.2024.00013(38-45)Online publication date: 15-Jul-2024
https://doi.org/10.1109/MobileCloud62079.2024.00013
Show More Cited By

Index Terms

Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatment
1. Applied computing
  1. Life and medical sciences
2. Computing methodologies
  1. Artificial intelligence
    1. Control methods

Recommendations

A Fuzzy Inference System for Closed-Loop Deep Brain Stimulation in Parkinson's Disease

Parkinsons disease is a complex neurodegenerative disorder for which patients present many symptoms, tremor being the main one. In advanced stages of the disease, Deep Brain Stimulation is a generalized therapy which can significantly improve the motor ...
Dissociation of Habit-Learning in Parkinson's and Cerebellar Disease

Damage to the medial-temporal region is known to result in declarative (explicit) memory deficits but nondeclarative (implicit) memory is largely unaffected by such lesions. Earlier studies have shown that some forms of implicit learning depend on ...
Startle habituation and midfrontal theta activity in parkinson disease

The ability to adapt to aversive stimuli is critical for mental health. Here, we investigate the relationship between habituation to startling stimuli and startle-related activity in medial frontal cortex as measured by EEG in both healthy control ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICCPS '23: Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023)

May 2023

291 pages

ISBN:9798400700361

DOI:10.1145/3576841

General Chairs:
Sayan Mitra
University of Illinois at Urbana Champaign
,
Nalini Venkatasubramanian
University of California at Irvine
,
Program Chairs:
Abhishek Dubey
Vanderbilt University
,
Lu Feng
University of Virginia
,
Publications Chairs:
Mahsa Ghasemi
Purdue University
,
Jonathan Sprinkle
Vanderbilt University

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGBED: ACM Special Interest Group on Embedded Systems
IEEE TCRTS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 May 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks
NIH
Medtronic PLC
Rune Labs

Conference

ICCPS '23

Sponsor:

SIGBED

ICCPS '23: ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023)

May 9 - 12, 2023

TX, San Antonio, USA

Acceptance Rates

Overall Acceptance Rate 25 of 91 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
513
Total Downloads

Downloads (Last 12 months)259
Downloads (Last 6 weeks)45

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pan MSchrum MMyers VBiyik EDragan ASalakhutdinov RKolter ZHeller KWeller AOliver NScarlett JBerkenkamp F(2024)Coprocessor actor criticProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3693662(39292-39307)Online publication date: 21-Jul-2024
https://dl.acm.org/doi/10.5555/3692070.3693662
Gao GYang XChi MWooldridge MDy JNatarajan S(2024)Get a head startProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i11.29102(12136-12144)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i11.29102
Mehregan JYuan XBagwe GJackson JMohammed EZhang LYu C(2024)Enhancing Adaptive Deep Brain Stimulation via Efficient Reinforcement Learning2024 IEEE Intelligent Mobile Computing (MobileCloud)10.1109/MobileCloud62079.2024.00013(38-45)Online publication date: 15-Jul-2024
https://doi.org/10.1109/MobileCloud62079.2024.00013
Hsu HMeng HLuo SDong JTarokh VPajic M(2024)REFORMA: Robust REinFORceMent Learning via Adaptive Adversary for Drones Flying under Disturbances2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10611002(5169-5175)Online publication date: 13-May-2024
https://doi.org/10.1109/ICRA57147.2024.10611002
Hsu HGao QPajic M(2024)ϵ-Neural Thompson Sampling of Deep Brain Stimulation for Parkinson Disease Treatment2024 ACM/IEEE 15th International Conference on Cyber-Physical Systems (ICCPS)10.1109/ICCPS61052.2024.00027(224-234)Online publication date: 13-May-2024
https://doi.org/10.1109/ICCPS61052.2024.00027

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents