Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation
Authors: 
Yi Lu, Shenghao Ren, Qiu Shen, Xun CaoAuthors Info & Claims
Pages 8932 - 8941
Abstract
Whole-body motion imitation has gained wide attention in recent years as it can enhance the locomotive capabilities of humanoid robots. In this task, non-intrusive human motion capturing with RGB cameras is commonly used for its low-cost, efficiency, portability and user-friendliness. However, RGB based methods always faces the problem of depth ambiguity, leading to inaccurate and unstable imitation. Accordingly, we propose to introduce pressure sensor into the non-intrusive humanoid motion imitation system for two considerations: first, pressure can be used to estimate the contact relationship and interaction force between human and the ground, which play a key role in the balancing and stabilizing motion; second, pressure can be measured in the manner of almost non-intrusive approach, which can keep the experience of human demonstrator. In this paper, we establish a RGB-Pressure (RGB-P) based humanoid imitation system, achieving accurate and stable end-to-end mapping from human body models to robot control parameters. Specifically, we use RGB camera to capture human posture and pressure insoles to measure the underfoot pressure during the movements of human demonstrator. Then, a constraint relationship between pressure and pose is studied to refine the estimated pose according to the support modes and balance mechanism, thereby enhancing consistency between human and robot motions. Experimental results demonstrate that fusing RGB and pressure can enhance overall robot motion execution performance by improving stability while maintaining imitation similarity.
References
[1]
2024. Vicon Systems. https://www.vicon.com/hardware/cameras/.
[2]
Firas Abi-Farrajl, Bernd Henze, Alexander Werner, Michael Panzirsch, Christian Ott, and Máximo A Roa. 2018. Humanoid teleoperation using task-relevant haptic feedback. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5010--5017.
[3]
Luis Almeida, Paulo Menezes, and Jorge Dias. 2022. Telepresence social robotics towards co-presence: A review. Applied Sciences 12, 11 (2022), 5557.
[4]
Ko Ayusawa and Eiichi Yoshida. 2017. Motion retargeting for humanoid robots based on simultaneous morphing parameter identification and motion optimiza tion. IEEE Transactions on Robotics 33, 6 (2017), 1343--1357.
[5]
ArchanaBalmik, ArnabPaikaray, Mrityunjay Jha, and Anup Nandy. 2022. Motion recognition using deep convolutional neural network for Kinect-based NAO teleoperation. Robotica 40, 9 (2022), 3233--3253.
[6]
FedericaBogo,AngjooKanazawa,ChristophLassner,PeterGehler,JavierRomero, and Michael J Black. 2016. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part V 14. Springer, 561--578.
[7]
Thomas Buschmann. 2010. Simulation and control of biped walking robots. Ph.D. Dissertation. Technische Universität München.
[8]
Jean Chagas Vaz, Dylan Wallace, and Paul Y Oh. 2021. Humanoid loco manipulation of pushed carts utilizing virtual reality teleoperation. In ASME International Mechanical Engineering Congress and Exposition, Vol. 85628. Ameri can Society of Mechanical Engineers, V07BT07A027.
[9]
StefanoDafarra,UgoPattacini,GiulioRomualdi,LorenzoRapetti,RiccardoGrieco, Kourosh Darvish, Gianluca Milani, Enrico Valli, Ines Sorrentino, Paolo Maria Viceconte, et al. 2024. iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robots. Science Robotics 9, 86 (2024), eadh3834.
[10]
D Dajles, F Siles, et al. 2018. Teleoperation of a humanoid robot using an optical motion capture system. In 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI). IEEE, 1--8.
[11]
Kourosh Darvish, Luigi Penco, Joao Ramos, Rafael Cisneros, Jerry Pratt, Eiichi Yoshida, Serena Ivaldi, and Daniele Pucci. 2023. Teleoperation of humanoid robots: A survey. IEEE Transactions on Robotics 39, 3 (2023), 1706--1727.
[12]
Kourosh Darvish, Yeshasvi Tirupachuri, Giulio Romualdi, Lorenzo Rapetti, Diego Ferigo, Francisco Javier Andrade Chavez, and Daniele Pucci. 2019. Whole-body geometric retargeting for humanoid robots. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). IEEE, 679--686.
[13]
Mohamed Elobaid, Yue Hu, Giulio Romualdi, Stefano Dafarra, Jan Babic, and Daniele Pucci. 2020. Telexistence and teleoperation for walking humanoid robots. In Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2. Springer, 1106--1121.
[14]
Xinyang Fan, Xin Shu, Baoxu Tu, Changyuan Liu, Fenglei Ni, and Zainan Jiang. 2023. Ahumanoidrobotteleoperationapproachbasedonwaist--armcoordination. Industrial Robot: the international journal of robotics research and application 50, 5 (2023), 804--813.
[15]
David Gouaillier, Vincent Hugel, Pierre Blazevic, Chris Kilner, Jerome Monceaux, Pascal Lafourcade, Brice Marnier, Julien Serre, and Bruno Maisonnier. [n.d.]. The nao humanoid: a combination of performance and affordability. ([n.d.]).
[16]
Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. 2024. Learning Human-to-Humanoid Real-Time Whole-Body Teleoperation. arXiv preprint arXiv:2403.04436 (2024).
[17]
Kai Hu, Christian Ott, and Dongheui Lee. 2014. Online human walking imitation in taskandjointspacebasedonquadraticprogramming.In2014IEEEInternational Conference on Robotics and Automation (ICRA). IEEE, 3458--3464.
[18]
Yasuhiro Ishiguro, Kunio Kojima, Fumihito Sugai, Shunichi Nozawa, Yohei Kaki uchi, Kei Okada, and Masayuki Inaba. 2018. High speed whole body dynamic motion experiment with real time master-slave humanoid robot system. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5835 5841.
[19]
Yasuhiro Ishiguro, Tasuku Makabe, Yuya Nagamatsu, Yuta Kojio, Kunio Kojima, Fumihito Sugai, Yohei Kakiuchi, Kei Okada, and Masayuki Inaba. 2020. Bilateral humanoid teleoperation system using whole-body exoskeleton cockpit TABLIS. IEEE Robotics and Automation Letters 5, 4 (2020), 6419--6426.
[20]
Matthew Johnson, Brandon Shrewsbury, Sylvain Bertrand, Duncan Calvert, TingfanWu,DanielDuran,DouglasStephen,NathanMertins,JohnCarff,William Rifenburgh, et al. 2017. Team IHMC's lessons learned from the DARPA Robotics Challenge: Finding data in the rubble. Journal of Field Robotics 34, 2 (2017), 241--261.
[21]
Steven Jens Jorgensen, Michael W Lanighan, Sylvain S Bertrand, Andrew Watson, Joseph S Altemus, R Scott Askew, Lyndon Bridgwater, Beau Domingue, Charlie Kendrick, Jason Lee, et al. 2019. Deploying the nasa valkyrie humanoid for ied response: An initial approach and evaluation summary. In 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids). IEEE, 1--8.
[22]
Shuuji Kajita, Hirohisa Hirukawa, Kazuhito Yokoi, and Kensuke Harada. 2005. Humanoid robots. Ohmsha Ltd (2005), 3--1.
[23]
AngjooKanazawa,MichaelJBlack,DavidWJacobs,andJitendraMalik.2018. End to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7122--7131.
[24]
Jonas Koenemann, Felix Burget, and Maren Bennewitz. 2014. Real-time imita tion of human whole-body motions by humanoids. In 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2806--2812.
[25]
Nikolaos Kofinas. 2012. Forward and inverse kinematics for the NAO humanoid robot. Technical University of Crete (2012).
[26]
Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision. 2252--2261.
[27]
Lars Kunze, Nick Hawes, Tom Duckett, Marc Hanheide, and Tomás Krajník. 2018. Artificial intelligence for long-term robot autonomy: A survey. IEEE Robotics and Automation Letters 3, 4 (2018), 4023--4030.
[28]
Jie Lei, Mingli Song, Ze-Nian Li, and Chun Chen. 2015. Whole-body humanoid robot imitation with pose similarity evaluation. Signal Processing 108 (2015), 136--146.
[29]
Gaofeng Li, Qiang Li, Chenguang Yang, Yuan Su, Zuqiang Yuan, and Xinyu Wu. 2023. The classification and new trends of shared control strategies in telerobotic systems: A survey. IEEE Transactions on Haptics (2023).
[30]
Zhihao Li, Jianzhuang Liu, Zhensong Zhang, Songcen Xu, and Youliang Yan. 2022. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference on Computer Vision. Springer, 590--606.
[31]
Zongmian Li, Jiri Sedlar, Justin Carpentier, Ivan Laptev, Nicolas Mansard, and Josef Sivic. 2019. Estimating 3d motion and forces of person-object interactions from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8640--8649.
[32]
Han Liang, Yannan He, Chengfeng Zhao, Mutian Li, Jingya Wang, Jingyi Yu, and Lan Xu. 2023. Hybridcap: Inertia-aid monocular capture of challenging human motions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 1539--1548.
[33]
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: a skinned multi-person linear model. ACM Trans actions on Graphics (TOG) 34, 6 (2015), 1--16.
[34]
Jing Luo, Wei He, and Chenguang Yang. 2020. Combined perception, control, and learning for teleoperation: key technologies, applications, and challenges. Cognitive Computation and Systems 2, 2 (2020), 33--43.
[35]
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. 2019. AMASS: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision. 5442 5451.
[36]
Yoshihiko Nakamura. 1990. Advanced robotics: redundancy and optimization. Addison-Wesley Longman Publishing Co., Inc.
[37]
Jaesung Oh, Inho Lee, Hyobin Jeong, and Jun-Ho Oh. 2019. Real-time humanoid whole-body remote control framework for imitating human motion based on kinematic mapping and motion constraints. Advanced Robotics 33, 6 (2019), 293--305.
[38]
Kazuya Otani and Karim Bouyarmane. 2017. Adaptive whole-body manipulation in human-to-humanoid multi-contact motion retargeting. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids). IEEE, 446--453.
[39]
Yongsheng Ou, Jianbing Hu, Zhiyang Wang, Yiqun Fu, Xinyu Wu, and Xiaoyun Li. 2015. A real-time human imitation system using kinect. International Journal of Social Robotics 7 (2015), 587--600.
[40]
Shaohua Pan, Qi Ma, Xinyu Yi, Weifeng Hu, Xiong Wang, Xingkang Zhou, Jijunnan Li, and Feng Xu. 2023. Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture. In SIGGRAPH Asia 2023 Conference Papers. 1--11.
[41]
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. 2019. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 10975--10985.
[42]
Luigo Penco, Brice Clément, Valerio Modugno, E Mingo Hoffman, Gabriele Nava, Daniele Pucci, Nikos G Tsagarakis, J-B Mouret, and Serena Ivaldi. 2018. Robust real-time whole-body motion retargeting from human to humanoid. In 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids). IEEE, 425--432.
[43]
Luigi Penco, Nicola Scianca, Valerio Modugno, Leonardo Lanari, Giuseppe Oriolo, and Serena Ivaldi. 2019. A multimode teleoperation framework for humanoid loco-manipulation: An application for the icub robot. IEEE Robotics & Automation Magazine 26, 4 (2019), 73--82.
[44]
Davis Rempe, Leonidas J Guibas, Aaron Hertzmann, Bryan Russell, Ruben Ville gas, and Jimei Yang. 2020. Contact and human dynamics from monocular video. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part V 16. Springer, 71--87.
[45]
Emily-Jane Rolley-Parnell, Dimitrios Kanoulas, Arturo Laurenzi, Brian Delhaisse, Leonel Rozo, Darwin G Caldwell, and Nikos G Tsagarakis. 2018. Bi-manual articulated robot teleoperation using an external RGB-Drangesensor.In201815th International Conference on Control, Automation, Robotics and Vision (ICARCV)
[46]
Jesse Scott, Bharadwaj Ravichandran, Christopher Funk, Robert T Collins, and Yanxi Liu. 2020. From image to stability: Learning dynamics from human pose. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXIII 16. Springer, 536--554.
[47]
Mario Selvaggio, Marco Cognetti, Stefanos Nikolaidis, Serena Ivaldi, and Bruno Siciliano. 2021. Autonomy in physical human-robot interaction: A brief survey. IEEE Robotics and Automation Letters 6, 4 (2021), 7989--7996.
[48]
Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. 2023. Deep Imitation Learning for Humanoid Loco manipulation through Human Teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids). IEEE, 1--8.
[49]
Soshi Shimada, Vladislav Golyanik, Weipeng Xu, Patrick Pérez, and Christian Theobalt. 2021. Neural monocular 3d human motion capture with physical awareness. ACM Transactions on Graphics (ToG) 40, 4 (2021), 1--15.
[50]
Soshi Shimada, Vladislav Golyanik, Weipeng Xu, and Christian Theobalt. 2020. Physcap: Physically plausible monocular 3d motion capture in real time. ACM Transactions on Graphics (ToG) 39, 6 (2020), 1--16.
[51]
SoftBank 2021. Nao document. SoftBank. http://doc.aldebaran.com/2--8/home_ nao.html.
[52]
Christopher Stanton, Anton Bogdanovych, and Edward Ratanasena. 2012. Teleop eration of a humanoid robot using full-body motion capture, example movements, and machine learning. In Proc. Australasian Conference on Robotics and Automa tion, Vol. 8. 51.
[53]
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep high-resolution representation learning for human poseestimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 5693--5703.
[54]
Susumu Tachi, Yasuyuki Inoue, and Fumihiro Kato. 2020. Telesar vi: Telexistence surrogate anthropomorphic robot vi. International Journal of Humanoid Robotics 17, 05 (2020), 2050019.
[55]
Yuchuang Tong, Haotian Liu, and Zhengtao Zhang. 2024. Advancements in Humanoid Robots: A Comprehensive Review and Future Prospects. IEEE/CAA Journal of Automatica Sinica 11, 2 (2024), 301--328.
[56]
Shashank Tripathi, Lea Müller, Chun-Hao P Huang, Omid Taheri, Michael J Black, and Dimitrios Tzionas. 2023. 3D human pose estimation via intuitive physics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4713--4725.
[57]
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. 2018. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV). 601--617.
[58]
Miomir Vukobratovic and Branislav Borovac. 2004. Zero-moment point-thirty f ive years of its life. International journal of humanoid robotics 1, 01 (2004), 157--173.
[59]
Feilong Wang, Furong Chen, Yanling Dong, Qi Yong, Xiaolong Yang, Long Zheng, Xinming Zhang, and Hang Su. 2023. Whole-Body Teleoperation Control of Dual-Arm Robot Using Sensor Fusion. Biomimetics 8, 8 (2023), 591.
[60]
Sen Wang, Xinxin Zuo, Runxiao Wang, Fuhua Cheng, and Ruigang Yang. 2017. A generative human-robot motion retargeting approach using a single depth sensor. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 5369--5376.
[61]
YeYuan,Shih-EnWei,TomasSimon,KrisKitani, andJason Saragih. 2021. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7159--7169.
[62]
Petrissa Zell, Bastian Wandt, and Bodo Rosenhahn. 2017. Joint 3d human motion capture and physical analysis from monocular videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 17--26.
[63]
Haodong Zhang, Weijie Li, Jiangpin Liu, Zexi Chen, Yuxiang Cui, Yue Wang, and Rong Xiong. 2022. Kinematic motion retargeting via neural latent optimization for learning sign language. IEEE Robotics and Automation Letters 7, 2 (2022), 4582--4589.
[64]
He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, and Xun Cao. 2024. MMVP: A Multimodal MoCapDataset with Vision and Pressure Sensors. arXiv preprint arXiv:2403.17610 (2024).
[65]
Liang Zhang, Zhihao Cheng, Yixin Gan, Guangming Zhu, Peiyi Shen, and Juan Song. 2016. Fast human whole body motion imitation algorithm for humanoid robots. In 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 1430--1435.
[66]
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. 2018. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 5628--5635.
[67]
Zhijun Zhang, Yaru Niu, Lingdong Kong, Shuyang Lin, and Hao Wang. 2019. A real-time upper-body robot imitation system. International Journal of Robotics and Control 2 (2019), 49--56.
[68]
Zhijun Zhang, Yaru Niu, Ziyi Yan, and Shuyang Lin. 2018. Real-time whole-body imitation by humanoid robots and task-oriented teleoperation using an analytical mapping method and quantitative evaluation. Applied Sciences 8, 10 (2018), 2005
Index Terms
- Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation
Recommendations
Whole-body motion imitation using human modeling
ROBIO '09: Proceedings of the 2008 IEEE International Conference on Robotics and BiomimeticsThe purpose of this study is to develop a methodology that enables a humanoid robot to imitate a whole body motion of a human. In the communication and interaction with a human being with motions and gestures, a humanoid robot needs not only to look ...
Whole-body humanoid robot imitation with pose similarity evaluation
Imitation is considered to be a kind of social learning that allows the transfer of information, actions, behaviors, etc. Whereas current robots are unable to perform as many tasks as human, it is a natural way for them to learn by imitations, just as ...
Visual Perception Method Based on Human Pose Estimation for Humanoid Robot Imitating Human Motions
CCRIS '21: Proceedings of the 2021 2nd International Conference on Control, Robotics and Intelligent SystemThe goal of the system proposed in this paper is to develop functions for humanoid robots to use vision to perceive human behavior and imitate human motions. With that aim, the position of the human joint points is recorded by a key point detector and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 57Total Downloads
- Downloads (Last 12 months)57
- Downloads (Last 6 weeks)27
Reflects downloads up to 13 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in