rPPG-HiBa:Hierarchical Balanced Framework for Remote Physiological Measurement
Authors: 
Yin Wang, Hao Lu, Ying-Cong Chen, Li Kuang, Mengchu Zhou, Shuiguang DengAuthors Info & Claims
Pages 2982 - 2991
Abstract
Remote photoplethysmography (rPPG) is a promising technique for non-contact physiological signal measurement. It has great potential applications in human health monitoring and emotion analysis. However, existing methods for the rPPG task ignore the long-tail phenomenon of physiological signal data, especially on multi-domain joint training. In addition, we find that the long-tail problem of the physiological label (phys-label) exists in different datasets, and the long-tail problem of some domain exists under the same phys-label. To tackle these problems, we propose a hierarchical balanced framework, to mitigate the bias caused by domain and phys-label imbalance. Specifically, we propose anti-spurious domain center learning tailored to learning domain-balanced embeddings space. Then, we adopt compact-aware continuity regularization to estimate phys-label-wise imbalances and construct continuity between embeddings. Extensive experiments demonstrate that our method outperforms the state-of-the-art in cross-dataset and intra-dataset settings. Our code is available at https://github.com/pywin/HiBa.
Supplemental Material
MP4 File - rPPG-HiBa:Hierarchical Balanced Framework for Remote Physiological Measurement
In the video we present the motivation for our study, the proposed framework and the experimental performance.
- Download
- 4.54 MB
References
[1]
Shin Ando and Chun Yuan Huang. 2017. Deep over-sampling framework for classifying imbalanced data. In ECML-PKDD. Springer, 770--785.
[2]
Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. 2019. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit Lett 124 (2019), 82--90.
[3]
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. 2018. A systematic study of the class imbalance problem in convolutional neural networks. IEEE trans. neural netw 106 (2018), 249--259.
[4]
Dong Cao, Xiangyu Zhu, Xingyu Huang, Jianzhu Guo, and Zhen Lei. 2020. Domain balancing: Face recognition on long-tailed domains. In Proc. IEEE CVPR. 5671--5679.
[5]
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. IEEE CVPR. 6299--6308.
[6]
Weixuan Chen and Daniel McDuff. 2018. DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. In Proc. ECCV. 349--365.
[7]
Wei-Hao Chung, Cheng-Ju Hsieh, Sheng-Hung Liu, and Chiou-Ting Hsu. 2022. Domain Generalized RPPG Network: Disentangled Feature Learning with Domain Permutation and Domain Augmentation. In Proc. ACCV. 807--823.
[8]
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. 2021. Parametric contrastive learning. In Proc. IEEE ICCV. 715--724.
[9]
Abhijit Das, Hao Lu, Hu Han, Antitza Dantcheva, Shiguang Shan, and Xilin Chen. 2021. BVPNet: Video-to-BVP Signal Prediction for Remote Heart Rate Estimation. In FG. IEEE, 01--08.
[10]
Gerard De Haan and Vincent Jeanne. 2013. Robust pulse rate from chrominancebased rPPG. IEEE Trans. Biomed. Eng. 60, 10 (2013), 2878--2886.
[11]
Gerard De Haan and Arno Van Leest. 2014. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiol. Meas. 35, 9 (2014), 1913.
[12]
Jingda Du, Si-Qi Liu, Bochao Zhang, and Pong C Yuen. 2023. Dual-Bridging With Adversarial Noise Generation for Domain Adaptive rPPG Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10355--10364.
[13]
Ming-Shiang Chen Gee-Sern Hsu, ArulMurugan Ambikapathi. 2017. Deep Learning with Time-Frequency Representation for Pulse Estimation. In Proc. IJCB. 642--650.
[14]
Shoufei Han, Kun Zhu, Mengchu Zhou, and Xinye Cai. 2022. Competition-driven multimodal multiobjective optimization and its application to feature selection for credit card fraud detection. IEEE Transactions on Systems, Man, and Cybernetics: Systems 52, 12 (2022), 7845--7857.
[15]
Shoufei Han, Kun Zhu, MengChu Zhou, Xiaojing Liu, Haoyue Liu, Yusuf Al- Turki, and Abdullah Abusorrah. 2022. A novel multiobjective fireworks algorithm and its applications to imbalanced distance minimization problems. IEEE/CAA Journal of Automatica Sinica 9, 8 (2022), 1476--1489.
[16]
Yunpei Jia, Jie Zhang, Shiguang Shan, and Xilin Chen. 2020. Single-side domain generalization for face anti-spoofing. In Proc. CVPR. 8484--8493.
[17]
Qi Kang, Lei Shi, MengChu Zhou, XueSongWang, QiDiWu, and ZhiWei. 2017. A distance-based weighted undersampling scheme for support vector machines and its application to imbalanced classification. IEEE transactions on neural networks and learning systems 29, 9 (2017), 4152--4165.
[18]
Zixuan Ke, Mohammad Kachuee, and Sungjin Lee. 2022. Domain-aware contrastive knowledge transfer for multi-domain imbalanced data. arXiv preprint arXiv:2204.01916 (2022).
[19]
Viktor Kessler, Patrick Thiam, Mohammadreza Amirian, and Friedhelm Schwenker. 2017. Pain recognition with camera photoplethysmography. In IEEE IPTA. 1--5.
[20]
Magdalena Lewandowska, Jacek Rumi'ski, Tomasz Kocejko, and J'drzej Nowak. 2011. Measuring pulse rate with a webcam'a non-contact method for evaluating cardiac activity. In FedCSIS. IEEE, 405--410.
[21]
Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. 2020. Multi-task temporal shift attention networks for on-device contactless vitals measurement. NeurIPS 33 (2020), 19400--19411.
[22]
Hao Lu, Hu Han, and S Kevin Zhou. 2021. Dual-GAN: Joint BVP and Noise Modeling for Remote Physiological Measurement. In Proc. IEEE CVPR. 12404-- 12413.
[23]
Hao Lu, Zitong Yu, Xuesong Niu, and Yingcong Chen. 2023. Neuron Structure Modeling for Generalizable Remote Physiological Measurement. arXiv preprint arXiv:2303.05955 (2023).
[24]
Daniel McDuff. 2022. Applications of camera-based physiological measurement beyond healthcare. In CVSM. Elsevier, 165--177.
[25]
Daniel McDuff. 2023. Camera measurement of physiological vital signs. Comput. Surveys 55, 9 (2023), 1--40.
[26]
Daniel McDuff, Sarah Gontarek, and Rosalind W Picard. 2014. Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Trans. Biomed. Eng. 61, 10 (2014), 2593--2601.
[27]
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2020. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 (2020).
[28]
Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. 2018. SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. In Proc. IEEE ICPR. 3580--3585.
[29]
Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. 2020. Rhythmnet: Endto- end heart rate estimation from face via spatial-temporal representation. IEEE Trans. on Image Process. 29 (2020), 2409--2423.
[30]
Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, and Guoying Zhao. 2020. Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling. In Proc. ECCV.
[31]
Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18, 10 (2010), 10762--10774.
[32]
Ying Qiu, Yang Liu, Juan Arteaga-Falconi, Haiwei Dong, and Abdulmotaleb El Saddik. 2019. EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video. IEEE Trans. Multimedia (2019).
[33]
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. 2022. Balanced mse for imbalanced visual regression. In Proc. IEEE CVPR. 7926--7935.
[34]
Ambareesh Revanur, Zhihua Li, Umur A Ciftci, Lijun Yin, and László A Jeni. 2021. The first vision for vitals (v4v) challenge for non-contact video-based physiological estimation. In Proc. CVPR workshop. 2760--2767.
[35]
Rencheng Song, Huan Chen, Juan Cheng, Chang Li, Yu Liu, and Xun Chen. 2021. PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE J-BHI (2021), 1--1.
[36]
Jeremy Speth, Nathan Vance, Patrick Flynn, and Adam Czajka. 2023. Noncontrastive unsupervised learning of physiological signals from video. In CVPR. 14464--14474.
[37]
Vojtech Franc, and Jirí Matas. 2018. Visual heart rate estimation with convolutional neural network. In Proc. BMVC. 3--6.
[38]
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. 2021. Density-based weighting for imbalanced regression. Machine Learning 110 (2021), 2187--2211.
[39]
Ronny Stricker, Steffen Müller, and Horst-Michael Gross. 2014. Non-contact video-based pulse rate measurement on a mobile service robot. In Proc. IEEE ISRHIC. 1056--1062.
[40]
Bo Sun, Qinglan Wei, Liandong Li, Qihua Xu, Jun He, and Lejun Yu. 2016. LSTM for dynamic emotion and group emotion recognition in the wild. In Proc. ACM ICMI. 451--457.
[41]
Weiyu Sun, Xinyu Zhang, Hao Lu, Ying Chen, Yun Ge, Xiaolin Huang, Jie Yuan, and Yingcong Chen. 2023. Resolve Domain Conflicts for Generalizable Remote Physiological Measurement. In Proceedings of the 31st ACM International Conference on Multimedia. 8214--8224.
[42]
Zhaodong Sun and Xiaobai Li. 2022. Contrast-Phys: Unsupervised Video-based Remote Physiological Measurement via Spatiotemporal Contrast. In Proc. ECCV.
[43]
Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. 2022. Invariant feature learning for generalized long-tailed classification. In ECCV. Springer, 709--726.
[44]
Luís Torgo, Rita P Ribeiro, Bernhard Pfahringer, and Paula Branco. 2013. Smote for regression. In Portuguese conference on artificial intelligence. Springer, 378--389.
[45]
Sergey Tulyakov, Xavier Alameda-Pineda, Elisa Ricci, Lijun Yin, Jeffrey F Cohn, and Nicu Sebe. 2016. Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In Proc. IEEE CVPR. 2396--2404.
[46]
Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. 2008. Remote plethysmographic imaging using ambient light. Opt. Express 16, 26 (2008), 21434--21445.
[47]
Kefan Wang, Jing An, Mengchu Zhou, Zhe Shi, Xudong Shi, and Qi Kang. 2022. Minority-weighted graph neural network for imbalanced node classification in social networks of internet of people. IEEE Internet of Things Journal 10, 1 (2022), 330--340.
[48]
Wenjin Wang, Albertus C den Brinker, Sander Stuijk, and Gerard de Haan. 2017. Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 64, 7 (2017), 1479--1491.
[49]
Wenjin Wang, Sander Stuijk, and Gerard De Haan. 2015. Exploiting spatial redundancy of image sensor for motion robust rPPG. IEEE Trans. Biomed. Eng. 62, 2 (2015), 415--425.
[50]
Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. 2022. Contrastive regression for domain adaptation on gaze estimation. In Proc. IEEE CVPR. 19376--19385.
[51]
Zhi-Kuan Wang, Ying Kao, and Chiou-Ting Hsu. 2019. Vision-Based Heart Rate Estimation Via A Two-Stream CNN. In Proc. IEEE ICIP. 3327--3331.
[52]
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. 2016. A discriminative feature learning approach for deep face recognition. In ECCV. Springer, 499--515.
[53]
Lin Xi, Weihai Chen, Changchen Zhao, Xingming Wu, and Jianhua Wang. 2020. Image enhancement for remote photoplethysmography in a low-light environment. In FG. IEEE, 1--7.
[54]
Shen Yan, Ziyan Zhao, Shixin Liu, and MengChu Zhou. 2023. BO-SMOTE: A Novel Bayesian-Optimization-Based Synthetic Minority Oversampling Technique. IEEE Transactions on Systems, Man, and Cybernetics: Systems (2023).
[55]
Yuzhe Yang, Hao Wang, and Dina Katabi. 2022. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In ECCV. Springer, 57--75.
[56]
Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. 2021. Delving into deep imbalanced regression. In ICML. PMLR, 11842--11851.
[57]
Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. 2021. Groupaware contrastive regression for action quality assessment. In Proc. IEEE ICCV. 7919--7928.
[58]
Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. 2019. Remote heart rate measurement from highly compressed facial videos: an end-toend deep learning solution with video enhancement. In Proc. IEEE ICCV. 151--160.
[59]
Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip Torr, and Guoying Zhao. 2022. PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer. IEEE CVPR (2022).
[60]
Nianyin Zeng, Xinyu Li, Peishu Wu, Han Li, and Xin Luo. 2024. A novel tensor decomposition-based efficient detector for low-altitude aerial objects with knowledge distillation scheme. IEEE/CAA Journal of Automatica Sinica 11, 2 (2024), 487--501.
[61]
Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, and Dina Katabi. 2024. Rank- N-Contrast: Learning Continuous Representations for Regression. Advances in Neural Information Processing Systems 36 (2024).
[62]
Honghao Zhu, MengChu Zhou, Yu Xie, and Aiiad Albeshri. 2024. A self-adapting and efficient dandelion algorithm and its application to feature selection for credit card fraud detection. IEEE/CAA Journal of Automatica Sinica 11, 2 (2024), 377--390.
Index Terms
- rPPG-HiBa:Hierarchical Balanced Framework for Remote Physiological Measurement
Recommendations
Resolve Domain Conflicts for Generalizable Remote Physiological Measurement
MM '23: Proceedings of the 31st ACM International Conference on MultimediaRemote photoplethysmography (rPPG) technology has become increasingly popular due to its non-invasive monitoring of various physiological indicators, making it widely applicable in multimedia interaction, healthcare, and emotion analysis. Existing rPPG ...
Intelligent Physiological Monitoring and Feedback System for Driver Safety
ICGEC '11: Proceedings of the 2011 Fifth International Conference on Genetic and Evolutionary ComputingIn the course of driving, sudden disease outbreak often cause serious traffic accidents. If drivers can real-time understand their physiological state, and then they could have a chance to get the best solution at the first time. Maybe accidents will be ...
Visual detection of short-wave blood pressure fluctuations
PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive EnvironmentsBlood pressure provides important information about a person’s state of health. It changes greatly during physical activity, but blood pressure at rest also indicates abnormal or pathological conditions. However, periodic fluctuations in blood pressure ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Key Research Project of Zhejiang Province
- National Science Foundation of China
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 53Total Downloads
- Downloads (Last 12 months)53
- Downloads (Last 6 weeks)53
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in