Hostname: page-component-cd9895bd7-jkksz Total loading time: 0 Render date: 2024-12-23T09:33:31.188Z Has data issue: false hasContentIssue false
  • English
  • Français

Motion generation for walking exoskeleton robot using multiple dynamic movement primitives sequences combined with reinforcement learning

Published online by Cambridge University Press:  07 January 2022

Peng Zhang  and
Junxia Zhang Open the ORCID record for Junxia Zhang [Opens in a new window]
Peng Zhang
Affiliation:
Tianjin University of Science and Technology, Dagunan Road, Tianjin, China Tianjin Key Laboratory for Integrated Design and Online Monitor Center of Light Design and Food Engineering Machinery Equipment, Tianjin, China
Junxia Zhang*
Affiliation:
Tianjin University of Science and Technology, Dagunan Road, Tianjin, China Tianjin Key Laboratory for Integrated Design and Online Monitor Center of Light Design and Food Engineering Machinery Equipment, Tianjin, China
*
*Corresponding author. E-mail: zjx@tust.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

In order to assist patients with lower limb disabilities in normal walking, a new trajectory learning scheme of limb exoskeleton robot based on dynamic movement primitives (DMP) combined with reinforcement learning (RL) was proposed. The developed exoskeleton robot has six degrees of freedom (DOFs). The hip and knee of each artificial leg can provide two electric-powered DOFs for flexion/extension. And two passive-installed DOFs of the ankle were used to achieve the motion of inversion/eversion and plantarflexion/dorsiflexion. The five-point segmented gait planning strategy is proposed to generate gait trajectories. The gait Zero Moment Point stability margin is used as a parameter to construct a stability criteria to ensure the stability of human-exoskeleton system. Based on the segmented gait trajectory planning formation strategy, the multiple-DMP sequences were proposed to model the generation trajectories. Meanwhile, in order to eliminate the effect of uncertainties in joint space, the RL was adopted to learn the trajectories. The experiment demonstrated that the proposed scheme can effectively remove interferences and uncertainties.

Keywords

dynamic movement primitiveswalking exoskeleton robotgait planning strategyreinforcement learning
Type
Research Article
Information
Robotica , Volume 40 , Issue 8 , August 2022 , pp. 2732 - 2747
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Kazerooni, H. and Steger, R., “The Berkeley lower extremity exoskeleton,” J. Dyn. Syst. Meas. Control 128(1), 14–25 (2006).CrossRefGoogle Scholar
Guizzo, E. and Goldstein, H., “The rise of the body bots,” IEEE Spectrum 42(10), 42 (2005).CrossRefGoogle Scholar
Huang, G. T., “Wearable robots,” Technol. Rev. 28(5), 70–73 (2004).Google Scholar
Walsh, C. J., Pasch, K. and Herr, H., “An Autonomous, Underactuated Exoskeleton for Load-Carrying Augmentation,” 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE, 2006) pp. 1410–1415.CrossRefGoogle Scholar
Walsh, C. J., “Biomimetic design of an under-actuated leg exoskeleton for load-carrying augmentation,” Massachusetts Inst of Tech Cambridge Media Lab (2006).CrossRefGoogle Scholar
Walsh, C. J., Endo, K. and Herr, H., “A quasi-passive leg exoskeleton for load-carrying augmentation,” Int. J. Humanoid Rob. 4(3), 487–506 (2007).CrossRefGoogle Scholar
Sankai, Y., “HAL: Hybrid assistive limb based on cybernics,” Rob. Res. 1(66), 25–34 (2010).CrossRefGoogle Scholar
Esquenazi, A., Talaty, M., Packel, A. and Saulino, M., “The re walk powered exoskeleton to restore ambulatory function to individuals with thoracic-level motor-complete spinal cord injury,” Am. J. Phys. Med. Rehabil. 91(11), 911–921 (2012).CrossRefGoogle Scholar
Hu, B. H., Krausz, N. E. and Hargrove, L. J., “A Novel Method for Bilateral Gait Segmentation Using a Single Thigh-Mounted Depth Sensor and IMU,” IEEE International Conference on Biomedical Robotics and Biomechanics, Enschede (2018) pp. 807–812.Google Scholar
Chao, K. and Hur, P., “A Step Towards Generating Human-Like Walking Gait Via Trajectory Optimization Through Contact for a Bipedal Robot with One-Sided Springs on Toes,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC (2017) pp. 4848–4853.Google Scholar
Faraji, S. and Ijspeert, A. J., “Scalable Closed-Form Trajectories for Periodic and Non-Periodic Human-Like Walking,” International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada (2019) pp. 5295–5301.Google Scholar
Wang, F., Wang, Y., Wen, S. and Zhao, S., “Nao Humanoid Robot Gait Planning Based on the Linear Inverted Pendulum,” Chinese Control and Decision Conference (CCDC), Taiyuan (2012) pp. 986–990.Google Scholar
Kasaei, M., Lau, N. and Pereira, A., “A Fast and Stable Omnidirectional Walking Engine for the Nao Humanoid Robot,” Robot World Cup XXIII (2019) pp. 99–111.Google Scholar
Sebastian, B., Ren, H. and Ben-Tzvi, P., “Neural Network Based Heterogeneous Sensor Fusion for Robot Motion Planning,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China (2019) pp. 2899–2904.Google Scholar
Demby, J., Gao, Y. and De Souza, G. N., “A Study on Solving the Inverse Kinematics of Serial Robots using Artificial Neural Network and Fuzzy Neural Network,” IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), New Orleans, LA, USA (2019) pp. 1–6.Google Scholar
Wang, H., Lu, T., Niu, B., Yan, H., Wang, X., Chen, J. and Li, Y., “Research on Fuzzy PID Control Algorithm for Lower Limb Rehabilitation Robot,” IEEE 4th Information Technology and Mechatronics Engineering Conference, Chongqing, China (2018) pp. 956–960.Google Scholar
Wen, Y. and Huiyi, L.. “Gait optimization of humanoid robot based on deep Q-network,” Comput. Modernizat. 56(4), 47–58 (2019).Google Scholar
Saputra, A. A., Botzheim, J., Sulistijono, I. A. and Kubota, N., “Biologically inspired control system for 3D locomotion of a humanoid biped robot,” IEEE Trans. Syst. Man Cybern. Syst. 7(46), 898–911 (2016).CrossRefGoogle Scholar
Ijspeert, A., Nakanishi, J. and Schaal, S.. “Movement Imitation with Nonlinear Dynamical Systems in Humanoid Robots,” IEEE International Conference on Robotics and Automation (ICRA2002) (2002) pp. 398–403.Google Scholar
Bottasso, C. L., Leonello, D. and Savini, B., “Path planning for autonomous vehicles by trajectory smoothing using motion primitives,” IEEE Trans. Control Syst. Technol. 16(6), 1152–1168 (2008).Google Scholar
Stulp, F., Theodorou, E. A. and Schaal, S., “Reinforcement learning with sequences of motion primitives for robust manipulation,” IEEE Trans. Robot. 28(6), 1360–1370 (2012).CrossRefGoogle Scholar
Theodorou, E., Buchli, J. and Schaal, S., “A generalized path integral control approach to reinforcement learning,” J. Mach. Learn. Res. 11(11), 3137–3181 (2010).Google Scholar
Nian, R., Liu, J. and Huang, B., “A review on reinforcement learning: Introduction and applications in industrial process control”, Comput. Chem. Eng. 139, 1–30 (2020).CrossRefGoogle Scholar
De Jesús Rubio, J., “Discrete time control based in neural networks for pendulums,” Appl. Soft Comput. 68(11), 821–832 (2018).CrossRefGoogle Scholar
Gao, X., Sun, B., Hu, X. and Zhu, K., “Echo state network for extended state observer and sliding mode control of vehicle drive motor with unknown hysteresis nonlinearity,” Math. Probl. Eng. 31(13), 1–13 (2020).Google Scholar
Zhao, J., “Neural networks-based optimal tracking control for nonzero-sum games of multi-player continuous-time nonlinear systems via reinforcement learning,” Neurocomputing 412(13), 167–176 (2020).CrossRefGoogle Scholar
Naeem, M., Rizvi, S. T. H. and Coronato, A., “A gentle introduction to reinforcement learning and its application in different fields”, IEEE Access 8(11), 209320–209344 (2020).CrossRefGoogle Scholar
Theodorou, E., Buchli, J. and Schaal, S., “Reinforcement Learning of Motor Skills in High Dimensions: A Path Integral Approach,” 2010 IEEE International Conference on Robotics and Automation (2010) pp. 2397–2403.Google Scholar
Luo, B., Liu, D., Huang, T. and Wang, D., “Model-free optimal tracking control via critic-only Q-learning,” IEEE Trans. Neural Networks Learn. Syst. 27(10), 2134–2144 (2016).CrossRefGoogle ScholarPubMed
Peng, J. and Williams, R. J., “Incremental multi-step Q-learning,” Mach. Learn. 22(7), 283–290 (1996).CrossRefGoogle Scholar
Lázaro-Camí, J. A., “The stochastic Hamilton-Jacobi equation,” J. Geom. Mech. 1(3), 295–315 (2008).CrossRefGoogle Scholar
Aphiratsakun, N., Chairungsarpsook, K. and Parnichkun, M., “ZMP Based Gait Generation of AIT’s Leg Exoskeleton,” 2010 The 2nd International Conference on Computer and Automation Engineering (ICCAE), vol. 5 (2010) pp. 886–890.Google Scholar