More Web Proxy on the site http://driver.im/

research-article

Multimodal In-bed Pose and Shape Estimation under the Blankets

Authors:

Joseph P. Robinson,

Yun FuAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 2411 - 2419

https://doi.org/10.1145/3503161.3548063

Published: 10 October 2022 Publication History

Abstract

Advancing technology to monitor our bodies and behavior while sleeping and resting are essential for healthcare. However, keen challenges arise from our tendency to rest under blankets. We present a multimodal approach to uncover the subjects and view bodies at rest without the blankets obscuring the view. For this, we introduce a channel-based fusion scheme to effectively fuse different modalities in a way that best leverages the knowledge captured by the multimodal sensors, including visual- and non-visual-based. The channel-based fusion scheme enhances the model's flexibility in the input at inference: one-to-many input modalities required at test time. Nonetheless, multimodal data or not, detecting humans at rest in bed is still a challenge due to the extreme occlusion when covered by a blanket. To mitigate the negative effects of blanket occlusion, we use an attention-based reconstruction module to explicitly reduce the uncertainty of occluded parts by generating uncovered modalities, which further update the current estimation via a cyclic fashion. Extensive experiments validate the proposed model's superiority over others.

References

[1]

Felix Achilles, Alexandru-Eugen Ichim, Huseyin Coskun, Federico Tombari, Soheyl Noachtar, and Nassir Navab. 2016. Patient MoCap: Human pose estimation under blanket occlusion for hospital monitoring applications. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Springer International Publishing, 491--499.

Digital Library

[2]

Karoon Agrawal and Neha Chauhan. 2012. Pressure ulcers: Back to the basics. Indian journal of plastic surgery: official publication of the Association of Plastic Surgeons of India, Vol. 45, 2 (2012), 244.

[3]

Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2014. 2d human pose estimation: New benchmark and state of the art analysis. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 3686--3693.

Digital Library

[4]

Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. 2016. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In Eur. Conf. Comput. Vis. Springer.

[5]

Leslie Casas, Nassir Navab, and Stefanie Demirci. 2019. Patient 3D body pose estimation from pressure imaging. International journal of computer assisted radiology and surgery, Vol. 14, 3 (2019), 517--524.

[6]

Kenny Chen, Paolo Gabriel, Abdulwahab Alasfour, Chenghao Gong, Werner K Doyle, Orrin Devinsky, Daniel Friedman, Patricia Dugan, Lucia Melloni, Thomas Thesen, et al. 2018. Patient-specific pose estimation in clinical environments. JTEHM, Vol. 6 (2018), 1--11.

[7]

Henry M Clever, Zackory Erickson, Ariel Kapusta, Greg Turk, Karen Liu, and Charles C Kemp. 2020. Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 6215--6224.

[8]

Qi Dang, Jianqin Yin, Bin Wang, and Wenqing Zheng. 2019. Deep learning based 2d human pose estimation: A survey. Tsinghua Science and Technology (2019).

[9]

John C Gower. 1975. Generalized procrustes analysis. Psychometrika, Vol. 40, 1 (1975), 33--51.

[10]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Identity mappings in deep residual networks. In Eur. Conf. Comput. Vis. Springer.

[11]

Mehrdad Heydarzadeh, Mehrdad Nourani, and Sarah Ostadabbas. 2016. In-bed posture classification using deep autoencoders. In International Conference of Engineering in Medicine and Biology Society. IEEE.

[12]

Chaoqun Hong, Jun Yu, Jian Wan, Dacheng Tao, and Meng Wang. 2015. Multimodal deep autoencoder for human pose recovery. IEEE Trans. Image Process., Vol. 24, 12 (2015), 5659--5670.

Digital Library

[13]

Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. 2018. End-to-end recovery of human shape and pose. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE.

[14]

Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. 2019. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In Int. Conf. Comput. Vis. IEEE.

[15]

Rui Li, Zhenyu Liu, and Jianrong Tan. 2019. A survey on 3D hand pose estimation: Cameras, methods, and datasets. Pattern Recognition, Vol. 93 (2019), 251--272.

Digital Library

[16]

Shuangjun Liu, Xiaofei Huang, Nihang Fu, Cheng Li, Zhongnan Su, and Sarah Ostadabbas. 2020. Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions. CoRR arXiv:2008.08735 (2020).

[17]

Shuangjun Liu and Sarah Ostadabbas. 2019. Seeing Under the Cover: A Physics Guided Learning Approach for In-Bed Pose Estimation. MICCAI (2019).

[18]

Shuangjun Liu, Yu Yin, and Sarah Ostadabbas. 2019. In-bed pose estimation: Deep learning with shallow dataset. Journal of Translational Engineering in Health and Medicine, Vol. 7 (2019), 1--12.

[19]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015a. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graph., Vol. 34, 6 (2015), 16 pages. https://doi.org/10.1145/2816795.2818013

[20]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015b. SMPL: A skinned multi-person linear model. ACM transactions on graphics (TOG), Vol. 34, 6 (2015), 1--16.

Digital Library

[21]

Roanna Lun and Wenbing Zhao. 2015. A survey of applications and human motion recognition with microsoft kinect. International Journal of Pattern Recognition and Artificial Intelligence, Vol. 29, 05 (2015), 1555008.

[22]

Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. 2017. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV). IEEE, 506--516.

[23]

Thomas B Moeslund and Erik Granum. 2001. A survey of computer vision-based human motion capture. Computer vision and image understanding, Vol. 81, 3 (2001), 231--268.

[24]

Thomas B Moeslund, Adrian Hilton, and Volker Krüger. 2006. A survey of advances in vision-based human motion capture and analysis. Computer vision and image understanding, Vol. 104, 2--3 (2006), 90--126.

[25]

Edna Bernal Monroy, Aurora Polo Rodr'iguez, Macarena Espinilla Estevez, and Javier Medina Quero. 2020. Fuzzy monitoring of in-bed postural changes for the prevention of pressure ulcers using inertial sensors attached to clothing. Journal of biomedical informatics, Vol. 107 (2020), 103476.

[26]

Tewodros Legesse Munea, Yalew Zelalem Jembre, Halefom Tekle Weldegebriel, Longbiao Chen, Chenxi Huang, and Chenhui Yang. 2020. The Progress of Human Pose Estimation: A Survey and Taxonomy of Models Applied in 2D Human Pose Estimation. IEEE Access, Vol. 8 (2020), 133330--133348.

[27]

Erik Murphy-Chutorian and Mohan Manubhai Trivedi. 2008. Head pose estimation in computer vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 31, 4 (2008), 607--626.

Digital Library

[28]

Xavier Perez-Sala, Sergio Escalera, Cecilio Angulo, and Jordi Gonzalez. 2014. A survey on model based approaches for 2D and 3D visual human pose recovery. Sensors, Vol. 14, 3 (2014), 4189--4210.

[29]

Barbara A Phillips, Jeffrey Okeson, Daniel Paesani, and Robin Gilmore. 1986. Effect of sleep position on sleep apnea and parafunctional activity. Chest, Vol. 90, 3 (1986), 424--429.

[30]

Maziyar Baran Pouyan, Javad Birjandtalab, Mehrdad Nourani, and MD Matthew Pompeo. 2016. Automatic limb identification and sleeping parameters assessment for pressure ulcer prevention. Computers in biology and medicine, Vol. 75 (2016), 98--108.

[31]

Weiyan Ren, Ou Ma, Hongxin Ji, and Xinyuan Liu. 2020. Human Posture Recognition Using a Hybrid of Fuzzy Logic and Machine Learning Approaches. IEEE Access, Vol. 8 (2020), 135628--135639.

[32]

Joseph P Robinson, Yuncheng Li, Ning Zhang, Yun Fu, and Sergey Tulyakov. 2019. Laplace landmark localization. In Int. Conf. Comput. Vis. IEEE, 10103--10112.

[33]

Ben Sapp and Ben Taskar. 2013. Modec: Multimodal decomposable models for human pose estimation. In IEEE Conf. Comput. Vis. Pattern Recog. 3674--3681.

Digital Library

[34]

Nikolaos Sarafianos, Bogdan Boteanu, Bogdan Ionescu, and Ioannis A Kakadiaris. 2016. 3d human pose estimation: A review of the literature and analysis of covariates. Computer Vision and Image Understanding, Vol. 152 (2016), 1--20.

Digital Library

[35]

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE.

[36]

Satoshi Tsutsumi, Hideo Ono, Yukimasa Yasumoto, and Hisato Ishii. 2019. Does the prone sleeping position affect the intracranial dural venous flow? Child's Nervous System, Vol. 35, 6 (2019), 913--916.

[37]

J Peter van Maanen, Kristel AW Meester, Lideke N Dun, Ioannis Koutsourelakis, Birgit I Witte, D Martin Laman, Antonius AJ Hilgevoord, and Nico de Vries. 2013. The sleep position trainer: a new treatment for positional obstructive sleep apnoea. Sleep and breathing, Vol. 17, 2 (2013), 771--779.

[38]

Yikai Wang, Wenbing Huang, Fuchun Sun, Tingyang Xu, Yu Rong, and Junzhou Huang. 2020. Deep multimodal fusion by channel exchanging. Adv. Neural Inform. Process. Syst., Vol. 33 (2020), 4835--4845.

[39]

Fan Yang, Ren Li, Georgios Georgakis, Srikrishna Karanam, Terrence Chen, Haibin Ling, and Ziyan Wu. 2020. Robust Multi-modal 3D Patient Body Modeling. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 86--95.

Digital Library

[40]

Linlin Yang, Shile Li, Dongheui Lee, and Angela Yao. 2019. Aligning latent spaces for 3d hand pose estimation. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE.

[41]

Wei Yang, Wanli Ouyang, Xiaolong Wang, Jimmy Ren, Hongsheng Li, and Xiaogang Wang. 2018. 3d human pose estimation in the wild by adversarial learning. In IEEE Conf. Comput. Vis. Pattern Recog. 5255--5264.

[42]

Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang. 2018. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. CoRR arXiv:1802.00124 (2018).

[43]

Mao Ye, Qing Zhang, Liang Wang, Jiejie Zhu, Ruigang Yang, and Juergen Gall. 2013. A survey on human motion analysis from depth data. In Time-of-flight and depth imaging. sensors, algorithms, and applications.

[44]

Rasoul Yousefi, Sarah Ostadabbas, Miad Faezipour, Masoud Farshbaf, Mehrdad Nourani, Lakshman Tamil, and Matthew Pompeo. 2011. Bed posture classification for pressure ulcer prevention. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, IEEE, 7175--7178.

[45]

Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. 2021. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Int. Conf. Comput. Vis. IEEE, 11446--11456.

[46]

Kun Zhou, Jinmiao Cai, Yao Li, Yulong Shi, Xiaoguang Han, Nianjuan Jiang, Kui Jia, and Jiangbo Lu. 2018. Adversarial 3D Human Pose Estimation via Multimodal Depth Supervision. CoRR arXiv:1809.07921 (2018).io

Cited By

Zhao LLin ZSun RWang A(2024)A Review of State-of-the-Art Methodologies and Applications in Action RecognitionElectronics10.3390/electronics1323473313:23(4733)Online publication date: 29-Nov-2024
https://doi.org/10.3390/electronics13234733
Wu ZXie FFang YLiang ZWan QXiong YCai X(2024)Seeing through the TactileProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596128:2(1-39)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659612
Liu CDong ZHuang LYan WWang XFang DChen X(2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643512
Show More Cited By

Index Terms

Multimodal In-bed Pose and Shape Estimation under the Blankets
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Shape inference

Recommendations

Privacy-Preserving in-bed pose monitoring: A fusion and reconstruction study
Abstract Background and objectives:
Recently, in-bed human pose estimation has attracted the interest of researchers due to its relevance to a wide range of healthcare applications. Compared to the general problem of human pose ...
Highlights
- We propose intermediate fusion techniques for in-bed pose estimation.
- We ...
Robust pose estimation in crowded scenes with direct pose-level inference
NIPS '21: Proceedings of the 35th International Conference on Neural Information Processing Systems

Multi-person pose estimation in crowded scenes is challenging because overlapping and occlusions make it difficult to detect person bounding boxes and infer pose cues from individual keypoints. To address those issues, this paper proposes a direct pose-...
Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities
Abstract
Over the past few years, research on animal pose estimation in computer vision field has grown in many aspects such as 2D and 3D pose estimation, 3D mesh reconstruction, and behavior prediction. Promoted by deep learning, more and more ...
Highlights
- Collecting and categorizing animal pose estimation work from 2013 to 2021 into 2D and 3D pose estimation, model reconstruction, and behavior prediction, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
239
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)20

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao LLin ZSun RWang A(2024)A Review of State-of-the-Art Methodologies and Applications in Action RecognitionElectronics10.3390/electronics1323473313:23(4733)Online publication date: 29-Nov-2024
https://doi.org/10.3390/electronics13234733
Wu ZXie FFang YLiang ZWan QXiong YCai X(2024)Seeing through the TactileProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596128:2(1-39)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659612
Liu CDong ZHuang LYan WWang XFang DChen X(2024)TagSleep3DProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435128:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643512
Yazici ZColantonio SEkenel H(2024)In-Bed Pose Estimation: A Review2024 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops59983.2024.10502680(154-158)Online publication date: 11-Mar-2024
https://doi.org/10.1109/PerComWorkshops59983.2024.10502680
Tandon AGoyal AClever HErickson Z(2024)BodyMAP - Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00240(2480-2489)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00240
Lei CLi FChen XYang L(2024)Pressure-Based In-Bed Pose Estimation Methods Using the Skinned Multi-person Linear Model2024 4th Asia Conference on Information Engineering (ACIE)10.1109/ACIE61839.2024.00019(74-79)Online publication date: 26-Jan-2024
https://doi.org/10.1109/ACIE61839.2024.00019
Zhu YXiao MXie YXiao ZJin GShuai L(2024)In-bed human pose estimation using multi-source information fusion for health monitoring in real-world scenariosInformation Fusion10.1016/j.inffus.2023.102209105(102209)Online publication date: May-2024
https://doi.org/10.1016/j.inffus.2023.102209
Kyrollos DFuller AGreenwood KHarrold JGreen J(2023)Under the Cover Infant Pose Estimation Using Multimodal DataIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.324422072(1-12)Online publication date: 2023
https://doi.org/10.1109/TIM.2023.3244220
Bigalke AHansen LDiesel JHennigs CRostalski PHeinrich M(2023)Anatomy-guided domain adaptation for 3D in-bed human pose estimationMedical Image Analysis10.1016/j.media.2023.10288789(102887)Online publication date: Oct-2023
https://doi.org/10.1016/j.media.2023.102887

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten