More Web Proxy on the site http://driver.im/

research-article

Incremental Real-Time Personalization in Human Activity Recognition Using Domain Adaptive Batch Normalization

Authors:

Alan Mazankiewicz,

Mario BergesAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 4, Issue 4

Article No.: 144, Pages 1 - 20

https://doi.org/10.1145/3432230

Published: 18 December 2020 Publication History

Abstract

Human Activity Recognition (HAR) from devices like smartphone accelerometers is a fundamental problem in ubiquitous computing. Machine learning based recognition models often perform poorly when applied to new users that were not part of the training data. Previous work has addressed this challenge by personalizing general recognition models to the unique motion pattern of a new user in a static batch setting. They require target user data to be available upfront. The more challenging online setting has received less attention. No samples from the target user are available in advance, but they arrive sequentially. Additionally, the motion pattern of users may change over time. Thus, adapting to new and forgetting old information must be traded off. Finally, the target user should not have to do any work to use the recognition system by, say, labeling any activities. Our work addresses all of these challenges by proposing an unsupervised online domain adaptation algorithm. Both classification and personalization happen continuously and incrementally in real time. Our solution works by aligning the feature distributions of all subjects, be they sources or the target, in hidden neural network layers. To this end, we normalize the input of a layer with user-specific mean and variance statistics. During training, these statistics are computed over user-specific batches. In the online phase, they are estimated incrementally for any new target user.

References

[1]

Zahraa Said Abdallah, Mohamed Medhat Gaber, Bala Srinivasan, and Shonali Krishnaswamy. 2015. Adaptive mobile activity recognition system with evolving data streams. Neurocomputing 150 (2015), 304--317.

Digital Library

[2]

Oresti Banos, Juan-Manuel Galvez, Miguel Damas, Hector Pomares, and Ignacio Rojas. 2014. Window size impact in human activity recognition. Sensors 14, 4 (2014), 6474--6499.

[3]

Paulo Barbosa, Kemilly Dearo Garcia, João Mendes-Moreira, and André CPLF de Carvalho. 2018. Unsupervised Domain Adaptation for Human Activity Recognition. In International Conference on Intelligent Data Engineering and Automated Learning. Springer, 623--630.

[4]

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning 79, 1-2 (2010), 151--175.

[5]

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. In Advances in neural information processing systems. 137--144.

[6]

Barbara Bruno, Fulvio Mastrogiovanni, Antonio Sgorbissa, Tullio Vernazza, and Renato Zaccaria. 2013. Analysis of human behavior recognition algorithms based on acceleration data. In 2013 IEEE International Conference on Robotics and Automation. IEEE, 1602--1607.

[7]

David M Burns and Cari M Whyne. 2020. Personalized Activity Recognition with Deep Triplet Embeddings. arXiv preprint arXiv:2001.05517 (2020).

[8]

Fabio Maria Cariucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, and Samuel Rota Bulo. 2017. Autodial: Automatic domain alignment layers. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 5077--5085.

[9]

Fabio Maria Carlucci, Lorenzo Porzi, Barbara Caputo, Elisa Ricci, and Samuel Rota Bulo. 2017. Just dial: Domain alignment layers for unsupervised domain adaptation. In International Conference on Image Analysis and Processing. Springer, 357--369.

[10]

Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A Systematic Study of Unsupervised Domain Adaptation for Robust Human-Activity Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.

Digital Library

[11]

Božidara Cvetković, Boštjan Kaluža, Matjaž Gams, and Mitja Luštrek. 2015. Adapting activity recognition to a person with Multi-Classifier Adaptive Training. Journal of Ambient Intelligence and Smart Environments 7, 2 (2015), 171--185.

Digital Library

[12]

Bozidara Cvetkovic, B Kaluza, M Luštrek, and Matjaz Gams. 2011. Semi-supervised learning for adaptation of human activity recognition classifier to the user. In Proc. of Workshop on Space, Time and Ambient Intelligence, IJCAI. Citeseer, 24--29.

[13]

Wan-Yu Deng, Qing-Hua Zheng, and Zhong-Min Wang. 2014. Cross-person activity recognition using reduced kernel extreme learning machine. Neural Networks 53 (2014), 1--7.

[14]

Tony Finch. 2009. Incremental calculation of weighted mean and variance. University of Cambridge 4, 11-5 (2009), 41--42.

[15]

Enrique Garcia-Ceja and Ramon Brena. 2015. Building personalized activity recognition models with scarce labeled data based on class similarities. In International conference on ubiquitous computing and ambient intelligence. Springer, 265--276.

[16]

Alexander Gepperth and Barbara Hammer. 2016. Incremental learning algorithms and applications.

[17]

Hristijan Gjoreski and Daniel Roggen. 2017. Unsupervised online activity discovery using temporal behaviour assumption. In Proceedings of the 2017 ACM International Symposium on Wearable Computers. 42--49.

Digital Library

[18]

Hirotaka Hachiya, Masashi Sugiyama, and Naonori Ueda. 2012. Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition. Neurocomputing 80 (2012), 93--101.

[19]

Jin-Hyuk Hong, Julian Ramos, and Anind K Dey. 2015. Toward personalized activity recognition systems with a semipopulation approach. IEEE Transactions on Human-Machine Systems 46, 1 (2015), 101--112.

[20]

Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (Lille, France) (ICML'15). JMLR.org, 448--456.

Digital Library

[21]

Artur Jordao, Antonio C Nazare Jr, Jessica Sena, and William Robson Schwartz. 2018. Human activity recognition based on wearable sensor data: A standardization of the state-of-the-art. arXiv preprint arXiv:1806.05226 (2018).

[22]

Nicholas D Lane, Ye Xu, Hong Lu, Shaohan Hu, Tanzeem Choudhury, Andrew T Campbell, and Feng Zhao. 2011. Enabling large-scale human activity inference on smartphones using community similarity networks (csn). In Proceedings of the 13th international conference on Ubiquitous computing. 355--364.

Digital Library

[23]

Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. 2016. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 (2016).

[24]

Jeffrey W Lockhart, Tony Pulickal, and Gary M Weiss. 2012. Applications of mobile activity recognition. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 1054--1058.

Digital Library

[25]

Takuya Maekawa and Shinji Watanabe. 2011. Unsupervised activity recognition with user's physical characteristics data. In 2011 15th Annual International Symposium on Wearable Computers. IEEE, 89--96.

Digital Library

[26]

Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. 2018. Robust place categorization with deep domain generalization. IEEE Robotics and Automation Letters 3, 3 (2018), 2093--2100.

[27]

Massimiliano Mancini, Hakan Karaoguz, Elisa Ricci, Patric Jensfelt, and Barbara Caputo. 2018. Kitting in the wild through online domain adaptation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1103--1109.

[28]

Massimiliano Mancini, Lorenzo Porzi, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. 2018. Boosting domain adaptation by discovering latent domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3771--3780.

[29]

Andrea Mannini and Stephen S Intille. 2018. Classifier personalization for activity recognition using wrist accelerometers. IEEE journal of biomedical and health informatics 23, 4 (2018), 1585--1594.

[30]

Nicholas A. Nystrom, Michael J. Levine, Ralph Z. Roskies, and J. Ray Scott. 2015. Bridges: A Uniquely Flexible HPC Resource for New Communities and Data Analytics. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure (St. Louis, Missouri) (XSEDE '15). ACM, New York, NY, USA, Article 30, 8 pages. https://doi.org/10.1145/2792745.2792775

[31]

Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2009), 1345--1359.

Digital Library

[32]

Juha Pärkkä, Luc Cluitmans, and Miikka Ermes. 2010. Personalization algorithm for real-time activity recognition using PDA, wireless motion bands, and binary decision tree. IEEE Transactions on Information Technology in Biomedicine 14, 5 (2010), 1211--1215.

Digital Library

[33]

Xin Qin, Yiqiang Chen, Jindong Wang, and Chaohui Yu. 2019. Cross-Dataset Activity Recognition via Adaptive Spatial-Temporal Transfer Learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1--25.

Digital Library

[34]

Sreenivasan Ramasamy Ramamurthy and Nirmalya Roy. 2018. Recent trends in machine learning for human activity recognition---A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8, 4 (2018), e1254.

[35]

Attila Reiss and Didier Stricker. 2013. Personalized mobile physical activity recognition. In Proceedings of the 2013 international symposium on wearable computers. 25--28.

Digital Library

[36]

Seyed Ali Rokni, Marjan Nourollahi, and Hassan Ghasemzadeh. 2018. Personalized human activity recognition using convolutional neural networks. In Thirty-Second AAAI Conference on Artificial Intelligence.

[37]

Ramyar Saeedi, Keyvan Sasani, Skyler Norgaard, and Assefaw H Gebremedhin. 2018. Personalized Human Activity Recognition using Wearables: A Manifold Learning-based Knowledge Transfer. In 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 1193--1196.

[38]

Sadiq Sani, Nirmalie Wiratunga, Stewart Massie, and Kay Cooper. 2017. kNN sampling for personalised human activity recognition. In International conference on case-based reasoning. Springer, 330--344.

[39]

Sadiq Sani, Nirmalie Wiratunga, Stewart Massie, and Kay Cooper. 2018. Matching networks for personalised human activity recognition. CEUR Workshop Proceedings.

[40]

Burr Settles. 2010. Active learning literature survey University of Wisconsin. Technical Report. Madison, Computer Science Technical Report 1648 52, 55-66 (2010), 11.

[41]

Hidetoshi Shimodaira. 2000. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of statistical planning and inference 90, 2 (2000), 227--244.

[42]

Pekka Siirtola and Juha Röning. 2019. Incremental Learning to Personalize Human Activity Recognition Models: The Importance of Human AI Collaboration. Sensors 19, 23 (2019), 5151.

[43]

Elnaz Soleimani and Ehsan Nazerfard. 2019. Cross-subject transfer learning in human activity recognition systems using generative adversarial networks. arXiv preprint arXiv:1903.12489 (2019).

[44]

Timo Sztyler and Heiner Stuckenschmidt. 2017. Online personalization of cross-subjects based activity recognition models on wearable devices. In 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 180--189.

[45]

Isaac Triguero, Salvador García, and Francisco Herrera. 2015. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems 42, 2 (2015), 245--284.

[46]

Niall Twomey, Tom Diethe, Xenofon Fafoutis, Atis Elsts, Ryan McConville, Peter Flach, and Ian Craddock. 2018. A comprehensive study of activity recognition using accelerometers. In Informatics, Vol. 5. Multidisciplinary Digital Publishing Institute, 27.

[47]

Jindong Wang, Vincent W Zheng, Yiqiang Chen, and Meiyu Huang. 2018. Deep transfer learning for cross-domain activity recognition. In proceedings of the 3rd International Conference on Crowd Science and Engineering. 1--8.

Digital Library

[48]

Gary Mitchell Weiss and Jeffrey Lockhart. 2012. The impact of personalization on smartphone-based activity recognition. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.

[49]

Gary M Weiss, Kenichi Yoneda, and Thaier Hayajneh. 2019. Smartphone and Smartwatch-Based Biometrics Using Activities of Daily Living. IEEE Access 7 (2019), 133190--133202.

[50]

Zhongtang Zhao, Yiqiang Chen, Junfa Liu, Zhiqi Shen, and Mingjie Liu. 2011. Cross-people mobile-phone based activity recognition. In Twenty-second international joint conference on artificial intelligence.

Cited By

Ye XSakurai KNair NWang K(2024)Machine Learning Techniques for Sensor-Based Human Activity Recognition with Data Heterogeneity—A ReviewSensors10.3390/s2424797524:24(7975)Online publication date: 13-Dec-2024
https://doi.org/10.3390/s24247975
Le HLakshminarayanan RLi JMishra VIntille S(2024)Collecting Self-reported Physical Activity and Posture Data Using Audio-based Ecological Momentary AssessmentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785848:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678584
Liu MBian SZhou BLukowicz PKostakos VKay JHoang T(2024)iKAN: Global Incremental Learning with KAN for Human Activity Recognition Across Heterogeneous DatasetsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676618(89-95)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676618
Show More Cited By

Index Terms

Incremental Real-Time Personalization in Human Activity Recognition Using Domain Adaptive Batch Normalization
1. Computing methodologies
  1. Machine learning
    1. Learning settings
      1. Online learning settings
    2. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Ubiquitous and mobile computing

Recommendations

Utilizing Self-Supervised Learning for Recognizing Human Activity in Older Adults through Labeling Applications in Real-World Smart Homes
GoodIT '24: Proceedings of the 2024 International Conference on Information Technology for Social Good

Deep learning models have significantly contributed to recognizing older adults’ daily activities for telemonitoring and assistance. However, recognizing human activities in real-world smart homes over the long term presents substantial challenges. ...
Few-Shot Human Activity Recognition on Noisy Wearable Sensor Data
Database Systems for Advanced Applications
Abstract
Most existing wearable sensor-based human activity recognition (HAR) models are trained on substantial labeled data. It is difficult for HAR to learn new-class activities unseen during training from a few samples. Very few researches of few-shot ...
Multiple-instance domain adaptation for cost-effective sensor-based human activity recognition
Abstract
Machine learning-based human activity recognition (HAR) is important as the means of human–computer interaction to empower the existing systems in many areas, such as healthcare, entertainment, logistics, and manufacturing. To build ...
Highlights
- Human activity recognition (HAR) and its application are beneficial in real-life.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 4, Issue 4

December 2020

1356 pages

EISSN:2474-9567

DOI:10.1145/3444864

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 December 2020

Published in IMWUT Volume 4, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
619
Total Downloads

Downloads (Last 12 months)100
Downloads (Last 6 weeks)8

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ye XSakurai KNair NWang K(2024)Machine Learning Techniques for Sensor-Based Human Activity Recognition with Data Heterogeneity—A ReviewSensors10.3390/s2424797524:24(7975)Online publication date: 13-Dec-2024
https://doi.org/10.3390/s24247975
Le HLakshminarayanan RLi JMishra VIntille S(2024)Collecting Self-reported Physical Activity and Posture Data Using Audio-based Ecological Momentary AssessmentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785848:3(1-35)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678584
Liu MBian SZhou BLukowicz PKostakos VKay JHoang T(2024)iKAN: Global Incremental Learning with KAN for Human Activity Recognition Across Heterogeneous DatasetsProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676618(89-95)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676618
Hong ZLi ZZhong SLyu WWang HDing YHe TZhang D(2024)CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised PretrainingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595978:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659597
Kianpisheh MMariakakis ATruong K(2024)exHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435008:1(1-30)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643500
Liu YSun LLiu Y(2024)Machine Learning-Based Human Activity Recognition Using Miniature Inertial and Magnetic Sensors2024 9th International Conference on Intelligent Computing and Signal Processing (ICSP)10.1109/ICSP62122.2024.10743490(934-941)Online publication date: 19-Apr-2024
https://doi.org/10.1109/ICSP62122.2024.10743490
Boulahia MBenharzallah STitouna FDilekh T(2024)Human Activity Recognition in Enhancing Healthcare for Aging Populations: Challenges, Innovations, and Future Directions2024 1st International Conference on Innovative and Intelligent Information Technologies (IC3IT)10.1109/IC3IT63743.2024.10869423(1-6)Online publication date: 3-Dec-2024
https://doi.org/10.1109/IC3IT63743.2024.10869423
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: 23-Jan-2024
https://dl.acm.org/doi/10.1109/COMST.2024.3357591
Haresamudram HEssa IPlötz T(2024)A Washing Machine is All You Need? On the Feasibility of Machine Data for Self-Supervised Human Activity Recognition2024 International Conference on Activity and Behavior Computing (ABC)10.1109/ABC61795.2024.10651688(1-10)Online publication date: 29-May-2024
https://doi.org/10.1109/ABC61795.2024.10651688
Ray AKolekar M(2024)Transfer learning and its extensive appositeness in human activity recognition: A surveyExpert Systems with Applications10.1016/j.eswa.2023.122538240(122538)Online publication date: Apr-2024
https://doi.org/10.1016/j.eswa.2023.122538
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents