More Web Proxy on the site http://driver.im/

research-article

Resolve Domain Conflicts for Generalizable Remote Physiological Measurement

Authors:

Yingcong ChenAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 8214 - 8224

https://doi.org/10.1145/3581783.3612265

Published: 27 October 2023 Publication History

Abstract

Remote photoplethysmography (rPPG) technology has become increasingly popular due to its non-invasive monitoring of various physiological indicators, making it widely applicable in multimedia interaction, healthcare, and emotion analysis. Existing rPPG methods utilize multiple datasets for training to enhance the generalizability of models. However, they often overlook the underlying conflict issues in the rPPG field, such as (1) label conflict resulting from different phase delays between physiological signal labels and face videos at the instance level, and (2) attribute conflict stemming from distribution shifts caused by head movements, illumination changes, skin types, etc. To address this, we introduce the DOmain-HArmonious framework (DOHA). Specifically, we first propose a harmonious phase strategy to eliminate uncertain phase delays and preserve the temporal variation of physiological signals. Next, we design a harmonious hyperplane optimization that reduces irrelevant attribute shifts and encourages the model's optimization towards a global solution that fits more valid scenarios. Our experiments demonstrate that DOHA significantly improves the performance of existing methods under multiple protocols.

References

[1]

Salma Alhagry, Aly Aly Fahmy, and Reda A El-Khoribi. Emotion recognition based on eeg using lstm recurrent neural network. International Journal of Advanced Computer Science and Applications, 8(10), 2017.

[2]

Serge Bobbia, Richard Macwan, Yannick Benezeth, Alamin Mansouri, and Julien Dubois. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters, 124:82--90, 2019.

Digital Library

[3]

Giuseppe Boccignone, Donatello Conte, Vittorio Cuculo, Alessandro D'Amelio, Giuliano Grossi, and Raffaella Lanzarotti. An open framework for remote-PPG methods and their assessment. IEEE Access, pages 1--1, 2020.

[4]

Tiffany Tianhui Cai, Jonathan Frankle, David J Schwab, and Ari S Morcos. Are all negatives created equal in contrastive instance discrimination? arXiv preprint arXiv:2010.06682, 2020.

[5]

Weixuan Chen and Daniel McDuff. Deepphys: Video-based physiological measurement using convolutional attention networks. In ECCV, 2018.

Digital Library

[6]

Wei-Hao Chung, Cheng-Ju Hsieh, Sheng-Hung Liu, and Chiou-Ting Hsu. Domain generalized rppg network: Disentangled feature learning with domain permutation and domain augmentation. In Proceedings of the Asian Conference on Computer Vision, pages 807--823, 2022.

[7]

Gerard De Haan and Vincent Jeanne. Robust pulse rate from chrominance-based rppg. IEEE Transactions on Biomedical Engineering, 60(10):2878--2886, 2013.

[8]

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. CoRR, abs/2010.01412, 2020.

[9]

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180--1189. PMLR, 2015.

Digital Library

[10]

Min Hu, Fei Qian, Dong Guo, Xiaohua Wang, Lei He, and Fuji Ren. Eta-rppgnet: Effective time-domain attention network for remote heart rate measurement. IEEE Transactions on Instrumentation and Measurement, 70:1--12, 2021.

[11]

Sinh Huynh, Rajesh Krishna Balan, Jeong Gil Ko, and Youngki Lee. Vitamon: Measuring heart rate variability using smartphone front camera. In Mi Zhang, editor, SenSys 2019 - Proceedings of the 17th Conference on Embedded Networked Sensor Systems, SenSys 2019 - Proceedings of the 17th Conference on Embedded Networked Sensor Systems, pages 1-14. Association for Computing Machinery, Nov. 2019. Publisher Copyright: © 2019 ACM.; 17th ACM Conference on Embedded Networked Sensor Systems, SenSys 2019; Conference date: 10-11-2019 Through 13-11-2019.

[12]

Seogkyu Jeon, Kibeom Hong, Pilhyeon Lee, Jewook Lee, and Hyeran Byun. Feature stylization and domain-aware contrastive learning for domain generalization. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, page 22--31, New York, NY, USA, 2021. Association for Computing Machinery.

Digital Library

[13]

Viktor Kessler, Patrick Thiam, Mohammadreza Amirian, and Friedhelm Schwenker. Pain recognition with camera photoplethysmography. In 2017 Seventh International Conference on Image Processing Theory, Tools and Applications, pages 1--5. IEEE, 2017.

[14]

Davis E King. Max-margin object detection. arXiv preprint arXiv:1502.00046, 2015.

[15]

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pages 5815--5826. PMLR, 2021.

[16]

Eugene Lee, Evan Chen, and Chen-Yi Lee. Meta-rppg: Remote heart rate estimation using a transductive meta-learner. In European Conference on Computer Vision, pages 392--409. Springer, 2020.

Digital Library

[17]

Magdalena Lewandowska, Jacek Rumi?ski, Tomasz Kocejko, and Jedrzej Nowak. Measuring pulse rate with a webcam - a non-contact method for evaluating cardiac activity. In 2011 Federated Conference on Computer Science and Information Systems, pages 405--410, 2011.

[18]

Buyu Li, Yu Liu, and Xiaogang Wang. Gradient harmonized single-stage detector. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8577--8584, 2019.

Digital Library

[19]

Bofan Lin, Xiaobai Li, Zitong Yu, and Guoying Zhao. Face liveness detection by rppg features and contextual patch-based cnn. In Proceedings of the 2019 3rd international conference on biometric engineering and applications, pages 61--68, 2019.

Digital Library

[20]

Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. Multi-task temporal shift attention networks for on-device contactless vitals measurement. Advances in Neural Information Processing Systems, 33:19400--19411, 2020.

[21]

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.

[22]

Hao Lu, Hu Han, and S Kevin Zhou. Dual-gan: Joint bvp and noise modeling for remote physiological measurement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12404--12413, 2021.

[23]

Hao Lu, Zitong Yu, Xuesong Niu, and Yingcong Chen. Neuron structure modeling for generalizable remote physiological measurement, 2023.

[24]

Ewa Magdalena Nowara, Tim K Marks, Hassan Mansour, and Ashok Veeraraghavan. Sparseppg: Towards driver monitoring using camera-based vital signs estimation in near-infrared. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1272--1281, 2018.

[25]

Lucas Mansilla, Rodrigo Echeveste, Diego H. Milone, and Enzo Ferrante. Domain generalization via gradient surgery, 2021.

[26]

Daniel McDuff. Applications of camera-based physiological measurement beyond healthcare. In Contactless Vital Signs Monitoring, pages 165--177. Elsevier, 2022.

[27]

Daniel McDuff. Camera measurement of physiological vital signs. ACM Computing Surveys, 55(9):1--40, 2023.

Digital Library

[28]

Jochen Meyer, Anastasia Kazakova, Merlin Büsing, and Susanne Boll. Visualization of complex health data on mobile devices. In Proceedings of the 2016 ACM Workshop on Multimedia for Personal Health and Health Care, pages 31--34, 2016.

Digital Library

[29]

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302.12400, 2023.

[30]

Xuesong Niu, Hu Han, Shiguang Shan, and Xilin Chen. Synrhythm: Learning a deep heart rate estimator from general to specific. In 2018 24th International Conference on Pattern Recognition, pages 3580--3585, 2018.

[31]

X. Niu, H. Han, S. Shan, and X. Chen. Vipl-hr: A multi-modal database for pulse estimation from less-constrained face video. 2018.

[32]

Xuesong Niu, Shiguang Shan, Hu Han, and Xilin Chen. Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation. IEEE Transactions on Image Processing, 29:2409--2423, 2020.

Digital Library

[33]

Xuesong Niu, Zitong Yu, Hu Han, Xiaobai Li, Shiguang Shan, and Guoying Zhao. Video-based remote physiological measurement via cross-verified feature disentangling. In European Conference on Computer Vision, 2020.

Digital Library

[34]

Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329, 2020.

[35]

Ming-Zher Poh, Daniel J McDuff, and Rosalind W Picard. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics express, 18(10):10762--10774, 2010.

[36]

Ming-Zher Poh, Daniel J. McDuff, and Rosalind W. Picard. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering, 58(1):7--11, 2011.

[37]

Rita Meziati Sabour, Yannick Benezeth, Pierre De Oliveira, Julien Chappe, and Fan Yang. Ubfc-phys: A multimodal database for psychophysiological studies of social stress. IEEE Transactions on Affective Computing, 2021.

[38]

Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.10745, 2018.

[39]

Robin P Smith, Jérôme Argod, Jean-Louis Pépin, and Patrick A Lévy. Pulse transit time: an appraisal of potential clinical applications. Thorax, 54(5):452--457, 1999.

[40]

Rencheng Song, Senle Zhang, Chang Li, Yunfei Zhang, Juan Cheng, and Xun Chen. Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks. IEEE Transactions on Instrumentation and Measurement, 69(10):7411--7421, 2020.

[41]

Radim ?petlík, Vojtech Franc, and Jirí Matas. Visual heart rate estimation with convolutional neural network. In Proceedings of the british machine vision conference, Newcastle, UK, pages 3--6, 2018.

[42]

Ronny Stricker, Steffen Müller, and Horst-Michael Gross. Non-contact video-based pulse rate measurement on a mobile service robot. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 1056--1062, 2014.

[43]

Bo Sun, Qinglan Wei, Liandong Li, Qihua Xu, Jun He, and Lejun Yu. Lstm for dynamic emotion and group emotion recognition in the wild. In Proceedings of the 18th ACM international conference on multimodal interaction, pages 451--457, 2016.

Digital Library

[44]

Zhaodong Sun and Xiaobai Li. Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast. In European Conference on Computer Vision, pages 492--510. Springer, 2022.

Digital Library

[45]

Chris Xing Tian, Haoliang Li, Xiaofei Xie, Yang Liu, and Shiqi Wang. Neuron coverage-guided domain generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1302--1311, 2022.

[46]

Yun-Yun Tsou, Yi-An Lee, Chiou-Ting Hsu, and Shang-Hung Chang. Siameserppg network: Remote photoplethysmography signal estimation from face videos. In Proceedings of the 35th annual ACM symposium on applied computing, pages 2066--2073, 2020.

Digital Library

[47]

Goran Udovi?i?, Jurica Ðerek, Mladen Russo, and Marjan Sikora. Wearable emotion recognition system based on gsr and ppg signals. In Proceedings of the 2nd international workshop on multimedia for personal health and health care, pages 53--59, 2017.

[48]

Wim Verkruysse, Lars O Svaasand, and J Stuart Nelson. Remote plethysmographic imaging using ambient light. Optics express, 16(26):21434--21445, 2008.

[49]

Jingye Wang, Ruoyi Du, Dongliang Chang, Kongming Liang, and Zhanyu Ma. Domain generalization via frequency-domain-based feature disentanglement and interaction. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, page 4821--4829, New York, NY, USA, 2022. Association for Computing Machinery.

Digital Library

[50]

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.

Digital Library

[51]

Mengzhu Wang, Jianlong Yuan, Qi Qian, Zhibin Wang, and Hao Li. Semantic data augmentation based distance metric learning for domain generalization. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, page 3214--3223, New York, NY, USA, 2022. Association for Computing Machinery.

Digital Library

[52]

Wenjin Wang, Albertus C Den Brinker, Sander Stuijk, and Gerard De Haan. Algorithmic principles of remote ppg. IEEE Transactions on Biomedical Engineering, 64(7):1479--1491, 2016.

[53]

Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. Eulerian video magnification for revealing subtle changes in the world. ACM transactions on graphics, 31(4):1--8, 2012.

[54]

Lin Xi, Weihai Chen, Changchen Zhao, Xingming Wu, and Jianhua Wang. Image enhancement for remote photoplethysmography in a low-light environment. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition, pages 1--7. IEEE, 2020.

Digital Library

[55]

Chenglin Yao, Shihe Wang, Jialu Zhang, Wentao He, Heshan Du, Jianfeng Ren, Ruibin Bai, and Jiang Liu. rppg-based spoofing detection for face mask attack using efficientnet on weighted spatial-temporal representation. In 2021 IEEE International Conference on Image Processing, pages 3872--3876. IEEE, 2021.

[56]

Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020.

[57]

Zitong Yu, Rizhao Cai, Zhi Li, Wenhan Yang, Jingang Shi, and Alex C Kot. Bench-marking joint face spoofing and forgery detection with visual and physiological cues. arXiv preprint arXiv:2208.05401, 2022.

[58]

Zitong Yu, Xiaobai Li, and Guoying Zhao. Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks. 2019.

[59]

Zitong Yu, Wei Peng, Xiaobai Li, Xiaopeng Hong, and Guoying Zhao. Remote heart rate measurement from highly compressed facial videos: An end-to-end deep learning solution with video enhancement. In 2019 IEEE/CVF International Conference on Computer Vision, pages 151--160, 2019.

[60]

Zitong Yu, Yuming Shen, Jingang Shi, Hengshuang Zhao, Philip Torr, and Guoying Zhao. Physformer: Facial video-based physiological measurement with temporal difference transformer. arXiv preprint arXiv:2111.12082, 2021.

[61]

Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012

Cited By

Wang JLu HHan HChen YHe DWu K(2025)Generalizable Remote Physiological Measurement via Semantic-Sheltered Alignment and Plausible Style RandomizationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.349705874(1-14)Online publication date: 2025
https://doi.org/10.1109/TIM.2024.3497058
Wang YLu HChen YKuang LZhou MDeng SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)rPPG-HiBa:Hierarchical Balanced Framework for Remote Physiological MeasurementProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680986(2982-2991)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680986
Zhang XSun WLu HChen YGe YHuang XYuan JChen Y(2024)Self-Similarity Prior Distillation for Unsupervised Remote Physiological MeasurementIEEE Transactions on Multimedia10.1109/TMM.2024.340572026(10290-10305)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3405720
Show More Cited By

Index Terms

Resolve Domain Conflicts for Generalizable Remote Physiological Measurement
1. Applied computing
  1. Life and medical sciences
    1. Health informatics

Recommendations

Visual detection of short-wave blood pressure fluctuations
PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

Blood pressure provides important information about a person’s state of health. It changes greatly during physical activity, but blood pressure at rest also indicates abnormal or pathological conditions. However, periodic fluctuations in blood pressure ...
Design and implementation of auscultation blood pressure measurement using vascular transit time and physiological parameters
2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
In this study, we have implemented an auscultation sphygmomanometer that can continuously estimate blood pressure via simultaneous measurement of phonocardiogram (PCG) and photoplethysmography (PPG) signals. In PCG, in the process of systole and diastole, ...
Exploring neural motion transfer for unsupervised remote physiological measurement: A practicality study
Abstract
Remote Photoplethysmography (rPPG) is a method to measure cardiac activity without the need for any contact-based sensors, garnering attention due to its non-invasive and convenient nature. Addressing the issue of substantial missing data in ...
Highlights
- The inaugural application of synthesized data in unsupervised remote physiological measurements.
- Unified resolution of motion artifact and sample diversity issues.
- In-depth analysis of physiological information within motion ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Jiangsu Provincial Social Development Key R&D Program

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
209
Total Downloads

Downloads (Last 12 months)142
Downloads (Last 6 weeks)4

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang JLu HHan HChen YHe DWu K(2025)Generalizable Remote Physiological Measurement via Semantic-Sheltered Alignment and Plausible Style RandomizationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.349705874(1-14)Online publication date: 2025
https://doi.org/10.1109/TIM.2024.3497058
Wang YLu HChen YKuang LZhou MDeng SCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)rPPG-HiBa:Hierarchical Balanced Framework for Remote Physiological MeasurementProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680986(2982-2991)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680986
Zhang XSun WLu HChen YGe YHuang XYuan JChen Y(2024)Self-Similarity Prior Distillation for Unsupervised Remote Physiological MeasurementIEEE Transactions on Multimedia10.1109/TMM.2024.340572026(10290-10305)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3405720
Zhang XSun WChen Y(2024)Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient HarmonizationIEEE Signal Processing Letters10.1109/LSP.2024.343004231(2595-2599)Online publication date: 2024
https://doi.org/10.1109/LSP.2024.3430042
Pirzada PWilde AHarris-Birtill D(2024)Remote Photoplethysmography for Heart Rate and Blood Oxygenation Measurement: A ReviewIEEE Sensors Journal10.1109/JSEN.2024.340541424:15(23436-23453)Online publication date: 1-Aug-2024
https://doi.org/10.1109/JSEN.2024.3405414
Xiao HLi ZXia ZLiu TZhou FAvolio A(2024)SimFuPulse: A self-similarity supervised model for remote photoplethysmography extraction from facial videosBiomedical Signal Processing and Control10.1016/j.bspc.2024.10673698(106736)Online publication date: Dec-2024
https://doi.org/10.1016/j.bspc.2024.106736
Li HLu HChen Y(2024)Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological MeasurementComputer Vision – ECCV 202410.1007/978-3-031-73247-8_21(356-374)Online publication date: 1-Nov-2024
https://doi.org/10.1007/978-3-031-73247-8_21

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents