More Web Proxy on the site http://driver.im/

research-article

Open access

APG: Audioplethysmography for Cardiac Monitoring in Hearables

Authors:

Richard Howard,

Longfei Shangguan,

Trausti ThormundssonAuthors Info & Claims

ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking

Article No.: 67, Pages 1 - 15

https://doi.org/10.1145/3570361.3613281

Published: 02 October 2023 Publication History

Abstract

This paper presents Audioplethysmography (APG), a novel cardiac monitoring modality for active noise cancellation (ANC) headphones. APG sends a low intensity ultrasound probing signal using an ANC headphone's speakers and receives the echoes via the on-board feedback microphones. We observed that, as the volume of ear canals slightly changes with blood vessel deformations, the heartbeats will modulate these ultrasound echoes. We built mathematical models to analyze the underlying physics and propose a multi-tone APG signal processing pipeline to derive the heart rate and heart rate variability in both constrained and unconstrained settings. APG enables robust monitoring of cardiac activities using mass-market ANC headphones in the presence of music playback and body motion such as running.

We conducted an eight-month field study with 153 participants to evaluate APG in various conditions. Our studies conform to the (Institutional Review Board) IRB policies from our company. The presented technology, experimental design, and results have been reviewed and further improved by feedback garnered from our internal Health Team, Product Team, User Experience (UX) Team and Legal team. Our results demonstrate that APG achieves consistently high HR (3.21% median error across 153 participants in all scenarios) and HRV (2.70% median error in interbeat interval, IBI) measurement accuracy. Our UX study further shows that APG is resilient to variation in: skin tone, sub-optimal seal conditions, and ear canal size.

References

[1]

Improving skin tone representation across google. Website, 2012.

[2]

Onyx ii model 9560 finger pulse oximeter. Website, 2012.

[3]

BOSE QC-35 Wireless Headphones. Website, 2020.

[4]

Gras 45bb kemar head & torso. Webpage, 2020.

[5]

Gras ra0404 prepolarized hi-res ear simulator. Webpage, 2020.

[6]

Polar h10 - the gold standard. Webpage, 2020.

[7]

A research space for earable computing. Webpage, 2020.

[8]

Samgsung Galaxy Buds. Website, 2020.

[9]

Wireless ecg, spo2, ptt and heart rate monitor reference design for medical and consumer wearables. Website, 2020.

[10]

Apple airpods. Website, 2021.

[11]

Earable: Helping the world sleep better. Webpage, 2022.

[12]

Find local maxima. Website, 2022.

[13]

Sawtooth wave. Website, 2022.

[14]

Zephyr™ performance systems. Website, 2022.

[15]

Apx58x b series audio analyzers. Website, 2023.

[16]

Developing the monk skin tone scale. Website, 2023.

[17]

Gras 12ba, 12be and 12bb ccp power modules with teds support. Website, 2023.

[18]

How many decibels does a human speak normally. Website, 2023.

[19]

Rme adi-2 pro fs r black edition. Website, 2023.

[20]

Rme fireface ufx+. Website, 2023.

[21]

Tascam series 8p dyna. Website, 2023.

[22]

Alberto P Avolio, Mark Butlin, and Andrew Walsh. Arterial blood pressure measurement and pulse wave analysis---their role in enhancing cardiovascular assessment. Physiological measurement, 2009.

[23]

Ananta Narayanan Balaji, Andrea Ferlini, Fahim Kawsar, and Alessandro Montanari. Stereo-bp: Non-invasive blood pressure sensing with earables. In Proceedings of the 24th International Workshop on Mobile Computing Systems and Applications, 2023.

Digital Library

[24]

Nam Bui, Nhat Pham, Jessica Jacqueline Barnitz, Zhanan Zou, Phuc Nguyen, Hoang Truong, Taeho Kim, Nicholas Farrow, Anh Nguyen, Jianliang Xiao, et al. ebp: A wearable system for frequent and comfortable blood pressure monitoring from user's ear. In The 25th annual international conference on mobile computing and networking, 2019.

[25]

Kayla-Jade Butkow, Ting Dang, Andrea Ferlini, Dong Ma, and Cecilia Mascolo. heart: Motion-resilient heart rate monitoring with in-ear microphones. arXiv preprint arXiv:2108.09393, 2021.

[26]

Ishan Chatterjee, Maruchi Kim, Vivek Jayaram, Shyamnath Gollakota, Ira Kemelmacher, Shwetak Patel, and Steven M Seitz. Clearbuds: wireless binaural earbuds for learning-based speech enhancement. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.

Digital Library

[27]

Romit Roy Choudhury. Earable computing: A new area to think about. In Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications, 2021.

Digital Library

[28]

Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, 2009.

[29]

Francis A Duck. Medical and non-medical protection standards for ultrasound and infrasound. Progress in biophysics and molecular biology, 2007.

[30]

Xiaoran Fan, Longfei Shangguan, Siddharth Rupavatharam, Yanyong Zhang, Jie Xiong, Yunfei Ma, and Richard Howard. Headfi: bringing intelligence to all headphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.

Digital Library

[31]

Xiaoran Fan and Trausti Thormundsson. Design earable sensing systems: Perspectives and lessons learned from industry. In EarComp, 2023.

[32]

Andrea Ferlini, Dong Ma, Robert Harle, and Cecilia Mascolo. Eargate: gait-based user identification with in-ear microphones. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, 2021.

Digital Library

[33]

Hannes Gamper, Sakari Tervo, and Tapio Lokki. Head orientation tracking using binaural headset microphones. In Audio Engineering Society Convention 131. Audio Engineering Society, 2011.

[34]

Francis Roosevelt Gilliam III, Robert Ciesielski, Karlen Shahinyan, Pratistha Shakya, John Cunsolo, Jal Mahendra Panchal, Bartlomiej Król-Józaga, Monika Król, Olivia Kierul, Charles Bridges, et al. In-ear infrasonic hemodynography with a digital health device for cardiovascular monitoring using the human audiome. npj Digital Medicine, 2022.

[35]

Jon D Hendrix, Jeffrey D Alderson, Chin Huang Yong, and Ryan A Hellman. Feedback howl management in adaptive noise cancellation system, 2019. US Patent 10,290,296.

[36]

Yincheng Jin, Yang Gao, Xiaotao Guo, Jun Wen, Zhengxiong Li, and Zhanpeng Jin. Earhealth: an earphone-based acoustic otoscope for detection of multiple ear diseases in daily life. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, 2022.

Digital Library

[37]

Fahim Kawsar, Chulhong Min, Akhil Mathur, Alessandro Montanari, Utku Günay Acer, and Marc Van den Broeck. esense: Open earable platform for human sensing. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, 2018.

Digital Library

[38]

Wolfgang Klippel. Nonlinear large-signal behavior of electrodynamic loudspeakers at low frequencies. Journal of the Audio Engineering Society, 1992.

[39]

Dong Ma, Andrea Ferlini, and Cecilia Mascolo. Oesense: employing occlusion effect for in-ear human sensing. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, 2021.

Digital Library

[40]

Monique Moderiano, Maureen McEvoy, Jessie Childs, and Adrian Esterman. Safety of ultrasound exposure: knowledge, attitudes, and practices of australasian sonographers. Journal of Diagnostic Medical Sonography, 2018.

[41]

Anh Nguyen, Raghda Alqurashi, Zohreh Raghebi, Farnoush Banaei-Kashani, Ann C Halbower, and Tam Vu. A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, 2016.

Digital Library

[42]

Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. Musicalheart: A hearty way of listening to music. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems, 2012.

Digital Library

[43]

Jun Nishimura and Tadahiro Kuroda. Eating habits monitoring using wireless wearable in-ear microphone. In 2008 3rd International Symposium on Wireless Pervasive Computing. IEEE, 2008.

[44]

International Non-Ionizing Radiation Committee of the International Radiation Protection Association et al. Interim guidelines on limits of human exposure to airborne ultrasound. Health Physics, 1984.

[45]

Sebastian Passler and Wolf-Joachim Fischer. Food intake activity detection using a wearable microphone system. In 2011 Seventh International Conference on Intelligent Environments. IEEE, 2011.

Digital Library

[46]

Nhat Pham, Tuan Dinh, Zohreh Raghebi, Taeho Kim, Nam Bui, Phuc Nguyen, Hoang Truong, Farnoush Banaei-Kashani, Ann Halbower, Thang Dinh, et al. Wake: a behind-the-ear wearable system for microsleep detection. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, 2020.

Digital Library

[47]

Tobias Röddiger, Christopher Clarke, Paula Breitling, Tim Schneegans, Haibin Zhao, Hans Gellersen, and Michael Beigl. Sensing with earables: A systematic literature review and taxonomy of phenomena. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.

Digital Library

[48]

Tobias Röddiger, Daniel Wolffram, David Laubenstein, Matthias Budde, and Michael Beigl. Towards respiration rate monitoring using an in-ear headphone inertial measurement unit. In EarComp, 2019.

Digital Library

[49]

Daniel Rouseff, John A Flynn, James A Ritcey, and Warren LJ Fox. Acoustic communication using time-reversal signal processing: Spatial and frequency diversity. In AIP Conference Proceedings. American Institute of Physics, 2004.

[50]

Fred Shaffer and Jay P Ginsberg. An overview of heart rate variability metrics and norms. Frontiers in public health, 2017.

[51]

SV Shilko, SP Salivonchik, et al. Calculation of pulse wave parameters with account of blood vessel deformation. Russian Journal of Biomechanics, 2001.

[52]

Robert R Sokal and Neal L Oden. Spatial autocorrelation in biology: 1. methodology. Biological journal of the Linnean Society, 1978.

[53]

Phyllis K Stein, Matthew S Bosner, Robert E Kleiger, and Brooke M Conger. Heart rate variability: a measure of cardiac autonomic tone. American heart journal, 1994.

[54]

Wei Sun, Franklin Mingzhe Li, Benjamin Steeper, Songlin Xu, Feng Tian, and Cheng Zhang. Teethtap: Recognizing discrete teeth gestures using motion and acoustic sensing on an earpiece. In 26th International Conference on Intelligent User Interfaces, 2021.

Digital Library

[55]

Xue Sun, Jie Xiong, Chao Feng, Wenwen Deng, Xudong Wei, Dingyi Fang, and Xiaojiang Chen. Earmonitor: In-ear motion-resilient acoustic sensing using commodity earphones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2023.

[56]

Conny MA van Ravenswaaij-Arts, Louis AA Kollee, Jeroen CW Hopman, Gerard BA Stoelinga, and Herman P van Geijn. Heart rate variability. Annals of internal medicine, 1993.

[57]

Joseph C Volpe Jr. Heart rate monitor for controlling entertainment devices, April 8 2008. US Patent 7,354,380.

[58]

Zi Wang, Yili Ren, Yingying Chen, and Jie Yang. Toothsonic: Earable authentication via acoustic toothprint. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022.

[59]

Metin C Yarici, Harry J Davies, Takashi Nakamura, Ian Williams, and Danilo P Mandic. Hearables: In-ear multimodal brain computer interfacing. In Brain-Computer Interface Research. Springer, 2021.

Cited By

Hu CMa XHuang XShen YMa D(2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699793
Mahmud SParikh VLiang QLi KZhang RAjit AGunda VAgarwal DGuimbretiere FZhang C(2024)ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the BodyProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997528:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699752
Han FYang PZuo YShang FXu FLi X(2024)EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech EnhancementProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785948:3(1-30)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678594
Show More Cited By

Index Terms

APG: Audioplethysmography for Cardiac Monitoring in Hearables
1. Hardware
  1. Emerging technologies
    1. Emerging interfaces
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Mobile computing

Recommendations

Pulse Arrival Time Based Cuff-Less and 24-H Wearable Blood Pressure Monitoring and its Diagnostic Value in Hypertension

Ambulatory blood pressure monitoring (ABPM) has become an essential tool in the diagnosis and management of hypertension. Current standard ABPM devices use an oscillometric cuff-based method which can cause physical discomfort to the patients with ...
Diabetic Macular Edema Screened by Handheld Smartphone-based Retinal Camera and Artificial Intelligence
Abstract
Our aim was to assess the tomographic presence of diabetic macular edema in type 2 diabetes patients screened for diabetic retinopathy with color fundus photographs and an artificial intelligence algorithm. Color fundus photographs obtained with a ...
Toward Hypertension Prediction Based on PPG-Derived HRV Signals: a Feasibility Study

Heart rate variability (HRV) is often used to assess the risk of cardiovascular disease, and data on this can be obtained via electrocardiography (ECG). However, collecting heart rate data via photoplethysmography (PPG) is now a lot easier. We ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ACM MobiCom '23: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking

October 2023

1605 pages

ISBN:9781450399906

DOI:10.1145/3570361

Chairs:
Xavier Costa,
Joerg Widmer,
Co-chairs:
Diego Perino,
Domenico Giustiniano,
Program Chair:
Haitham Al Hassanieh,
Program Co-chairs:
Arash Asadi,
Landon Cox

Copyright © 2023 Owner/Author(s).

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 October 2023

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACM MobiCom '23

Sponsor:

SIGMOBILE

ACM MobiCom '23: 29th Annual International Conference on Mobile Computing and Networking

October 2 - 6, 2023

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 440 of 2,972 submissions, 15%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
4,019
Total Downloads

Downloads (Last 12 months)2,975
Downloads (Last 6 weeks)1,493

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hu CMa XHuang XShen YMa D(2024)LR-Auth: Towards Practical Implementation of Implicit User Authentication on EarbudsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997938:4(1-27)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699793
Mahmud SParikh VLiang QLi KZhang RAjit AGunda VAgarwal DGuimbretiere FZhang C(2024)ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the BodyProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997528:4(1-32)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699752
Han FYang PZuo YShang FXu FLi X(2024)EarSpeech: Exploring In-Ear Occlusion Effect on Earphones for Data-efficient Airborne Speech EnhancementProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785948:3(1-30)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678594
Wang DZhang XWang KWang LFan XZhang Y(2024)RDGait: A mmWave Based Gait User Recognition System for Complex Indoor Environments Using Single-chip RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785528:3(1-31)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678552
Mahmud SAgarwal DAjit ALiang QViranda TGuimbretiere FZhang CKostakos VKay JHoang T(2024)MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on EyeglassesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676619(96-103)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676619
Parikh VMahmud SAgarwal DLi KGuimbretière FZhang CKostakos VKay JHoang T(2024)EchoGuide: Active Acoustic Guidance for LLM-Based Eating Event Analysis from Egocentric VideosProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676611(40-47)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676611
Christofferson KBhalla SCafazzo JMariakakis AKostakos VKay JHoang T(2024)On the Production and Measurement of Cardiac Sounds in the Ear CanalCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680526(685-690)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680526
Butkow KFerlini AKawsar FMascolo CMontanari AKostakos VKay JHoang T(2024)EarTune: Exploring the Physiology of Music ListeningCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3680519(644-649)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3680519
Liu XLiu HLi JYang ZHuang YZhang JKostakos VKay JHoang T(2024)AcousAF: Acoustic Sensing-Based Atrial Fibrillation Detection System for Mobile PhonesCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678488(377-383)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678488
Romero JFerlini ASpathis DDang TFarrahi KKawsar FMontanari AXu CDavies N(2024)OptiBreathe: An Earable-based PPG System for Continuous Respiration Rate, Breathing Phase, and Tidal Volume MonitoringProceedings of the 25th International Workshop on Mobile Computing Systems and Applications10.1145/3638550.3641136(99-106)Online publication date: 28-Feb-2024
https://dl.acm.org/doi/10.1145/3638550.3641136
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents