More Web Proxy on the site http://driver.im/

research-article

Smart-Badge: A wearable badge with multi-modal sensors for kitchen activity recognition

Authors:

Agnes Gruenerbl,

Paul LukowiczAuthors Info & Claims

UbiComp/ISWC '22 Adjunct: Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers

Pages 356 - 363

https://doi.org/10.1145/3544793.3560391

Published: 24 April 2023 Publication History

Abstract

Human health is closely associated with their daily behavior and environment. However, keeping a healthy lifestyle is still challenging for most people as it is difficult to recognize their living behaviors and identify their surrounding situations to take appropriate action. Human activity recognition is a promising approach to building a behavior model of users, by which users can get feedback about their habits and be encouraged to develop a healthier lifestyle. In this paper, we present a smart light wearable badge with six kinds of sensors, including an infrared array sensor MLX90640 offering privacy-preserving, low-cost, and non-invasive features, to recognize daily activities in a realistic unmodified kitchen environment. A multi-channel convolutional neural network (MC-CNN) based on data and feature fusion methods is applied to classify 14 human activities associated with potentially unhealthy habits. Meanwhile, we evaluate the impact of the infrared array sensor on the recognition accuracy of these activities. We demonstrate the performance of the proposed work to detect the 14 activities performed by ten volunteers with an average accuracy of 92.44 % and an F1 score of 88.27 %.

References

[1]

Antonio A Aguileta, Ramon F Brena, Oscar Mayora, Erik Molino-Minero-Re, and Luis A Trejo. 2019. Multi-sensor fusion for activity recognition—A survey. Sensors 19, 17 (2019), 3808.

[2]

Shubham Bansal, Shubham Khandelwal, Shubham Gupta, and Dushyant Goyal. 2013. Kitchen activity recognition based on scene context. In 2013 IEEE International Conference on Image Processing. IEEE, Melbourne, VIC, Australia, 3461–3465.

[3]

Lydia A Bazzano, Jiang He, Lorraine G Ogden, Catherine M Loria, Suma Vupputuri, Leann Myers, and Paul K Whelton. 2002. Fruit and vegetable intake and risk of cardiovascular disease in US adults: the first National Health and Nutrition Examination Survey Epidemiologic Follow-up Study. The American journal of clinical nutrition 76, 1 (2002), 93–99.

[4]

Pratool Bharti, Debraj De, Sriram Chellappan, and Sajal K Das. 2018. HuMAn: Complex activity recognition with multi-modal multi-positional body sensing. IEEE Transactions on Mobile Computing 18, 4 (2018), 857–870.

Digital Library

[5]

Sizhen Bian, Mengxi Liu, Bo Zhou, and Paul Lukowicz. 2022. The State-of-the-Art Sensing Techniques in Human Activity Recognition: A Survey. Sensors 22, 12 (2022), 4596.

[6]

Valentina Bianchi, Marco Bassoli, Gianfranco Lombardo, Paolo Fornacciari, Monica Mordonini, and Ilaria De Munari. 2019. IoT wearable sensor and deep learning: An integrated approach for personalized human activity recognition in a smart home environment. IEEE Internet of Things Journal 6, 5 (2019), 8553–8562.

[7]

Jingyuan Cheng, Oliver Amft, and Paul Lukowicz. 2010. Active capacitive sensing: Exploring a new wearable sensing modality for activity recognition. In International conference on pervasive computing. Springer, Berlin, Heidelberg, 319–336.

Digital Library

[8]

Yong Du, Wei Wang, and Liang Wang. 2015. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Boston, MA, USA, 1110–1118.

[9]

Raffaele Gravina, Parastoo Alinia, Hassan Ghasemzadeh, and Giancarlo Fortino. 2017. Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges. Information Fusion 35(2017), 68–80.

[10]

Raffaele Gravina and Qimeng Li. 2019. Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion. Information Fusion 48(2019), 1–10.

Digital Library

[11]

Lukas Günthermann, Lloyd Pellatt, and Daniel Roggen. 2022. Slow Feature Preprocessing in Deep Neural Networks for Wearable Sensor-Based Locomotion Recognition. In 2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, Pisa, Italy, 58–61.

[12]

Shuo Jiang, Qinghua Gao, Huaiyang Liu, and Peter B Shull. 2020. A novel, co-located EMG-FMG-sensing wearable armband for hand gesture recognition. Sensors and Actuators A: Physical 301 (2020), 111738.

[13]

Haruka Kamachi, Tahera Hossain, Fuyuka Tokuyama, Anna Yokokubo, and Guillaume Lopez. 2021. Prediction of eating activity using smartwatch. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers. ACM, Virtual, 304–309.

[14]

D Scott Kehler, Jacqueline L Hay, Andrew N Stammers, Naomi C Hamm, Dustin E Kimber, Annette SH Schultz, Andrea Szwajcer, Rakesh C Arora, Navdeep Tangri, and Todd A Duhamel. 2018. A systematic review of the association between sedentary behaviors with frailty. Experimental gerontology 114 (2018), 1–12.

[15]

Jinna Lei, Xiaofeng Ren, and Dieter Fox. 2012. Fine-grained kitchen activity recognition using rgb-d. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, Pittsburgh, USA, 208–211.

Digital Library

[16]

Fei Luo, Stefan Poslad, and Eliane Bodanese. 2019. Kitchen activity detection for healthcare using a low-power radar-enabled sensor network. In ICC 2019-2019 IEEE International Conference on Communications (ICC). IEEE, 1–7.

[17]

Hitoshi Matsuyama, Shunsuke Aoki, Takuro Yonezawa, Kei Hiroi, Katsuhiko Kaji, and Nobuo Kawaguchi. 2021. Deep Learning for Ballroom Dance Recognition: A Temporal and Trajectory-Aware Classification Model With Three-Dimensional Pose Estimation and Wearable Sensing. IEEE Sensors Journal 21, 22 (2021), 25437–25448.

[18]

Jean Mayer and Donald W Thomas. 1967. Regulation of Food Intake and Obesity: The regulation of food intake is complex; a number of abnormalities may cause obesity.Science 156, 3773 (1967), 328–337.

[19]

Yasser Mohammad, Kazunori Matsumoto, and Keiichiro Hoashi. 2017. A dataset for activity recognition in an unmodified kitchen using smart-watch accelerometers. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia. ACM, Stuttgart, Germany, 63–68.

Digital Library

[20]

Pradeep Natarajan and Ramakant Nevatia. 2008. Online, real-time tracking and recognition of human actions. In 2008 IEEE Workshop on Motion and video Computing. IEEE, 1–8.

Digital Library

[21]

George A Oguntala, Raed A Abd-Alhameed, Nazar T Ali, Yim-Fun Hu, James M Noras, Nnabuike N Eya, Issa Elfergani, and Jonathan Rodriguez. 2019. SmartWall: Novel RFID-enabled ambient human activity recognition using machine learning for unobtrusive health monitoring. IEEE Access 7(2019), 68022–68033.

[22]

Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.

[23]

Sen Qiu, Hongkai Zhao, Nan Jiang, Zhelong Wang, Long Liu, Yi An, Hongyu Zhao, Xin Miao, Ruichen Liu, and Giancarlo Fortino. 2022. Multi-sensor information fusion based on machine learning for real applications in human activity recognition: State-of-the-art and research challenges. Information Fusion 80(2022), 241–265.

Digital Library

[24]

Nora D Volkow, Gene-Jack Wang, and Ruben D Baler. 2011. Reward, dopamine and the control of food intake: implications for obesity. Trends in cognitive sciences 15, 1 (2011), 37–46.

[25]

Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019. Deep learning for sensor-based activity recognition: A survey. Pattern recognition letters 119 (2019), 3–11.

Digital Library

[26]

Cheng Xu, Duo Chai, Jie He, Xiaotong Zhang, and Shihong Duan. 2019. InnoHAR: A deep neural network for complex human activity recognition. Ieee Access 7(2019), 9893–9902.

[27]

Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. 2015. Deep convolutional neural networks on multichannel time series for human activity recognition. In Twenty-fourth international joint conference on artificial intelligence. ACM, Buenos Aires, Argentina, 3995–4001.

[28]

Xiaodong Yang and YingLi Tian. 2016. Super normal vector for human activity recognition with depth cameras. IEEE transactions on pattern analysis and machine intelligence 39, 5(2016), 1028–1039.

[29]

Xiaodong Yang, Chenyang Zhang, and YingLi Tian. 2012. Recognizing actions using depth motion maps-based histograms of oriented gradients. In Proceedings of the 20th ACM international conference on Multimedia. ACM, Nara, Japan, 1057–1060.

Digital Library

[30]

Shibo Zhang, Yuqi Zhao, Dzung Tri Nguyen, Runsheng Xu, Sougata Sen, Josiah Hester, and Nabil Alshurafa. 2020. Necksense: A multi-sensor necklace for detecting eating activities in free-living conditions. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 4, 2 (2020), 1–26.

Digital Library

[31]

Shangyue Zhu, Junhong Xu, Hanqing Guo, Qiwei Liu, Shaoen Wu, and Honggang Wang. 2018. Indoor human activity recognition based on ambient radar with signal processing and machine learning. In 2018 IEEE international conference on communications (ICC). IEEE, Kansas City, MO, USA, 1–6.

Cited By

Adam GTsangrassoulis A(2024)Design and Development of a Low-Cost Educational Platform for Investigating Human-Centric Lighting (HCL) SettingsComputers10.3390/computers1312033813:12(338)Online publication date: 14-Dec-2024
https://doi.org/10.3390/computers13120338
Sabbella HWeerakoon DGulati MMisra A(2024)PA2BLO: Low-Power, Personalized Audio Badge2024 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom59722.2024.10494427(154-163)Online publication date: 11-Mar-2024
https://doi.org/10.1109/PerCom59722.2024.10494427
Liu MSuh SVargas JZhou BGrünerbl ALukowicz P(2024)A Wearable Multi-modal Edge-Computing System for Real-Time Kitchen Activity RecognitionHuman Activity Recognition and Anomaly Detection10.1007/978-981-97-9003-6_9(132-145)Online publication date: 17-Nov-2024
https://doi.org/10.1007/978-981-97-9003-6_9
Show More Cited By

Recommendations

Wearable Sensor Badge and Sensor Jacket for Context Awareness
ISWC '99: Proceedings of the 3rd IEEE International Symposium on Wearable Computers

The addition of sensors to wearable computers allows them to adapt their functions to more suit the activities and situation of their wearers. Here a wearable sensor badge is described constructed from (hard) electronic components, which can sense ...
Activity recognition with hand-worn magnetic sensors

Activity recognition is a key technology for realizing ambient assisted living applications such as care of the elderly and home automation. This paper proposes a new activity recognition method that employs hand-worn magnetic sensors to recognize a ...
Recognizing multi-user activities using wearable sensors in a smart home

The advances of wearable sensors and wireless networks offer many opportunities to recognize human activities from sensor readings in pervasive computing. Existing work so far focuses mainly on recognizing activities of a single user in a home ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp/ISWC '22 Adjunct: Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers

September 2022

538 pages

ISBN:9781450394239

DOI:10.1145/3544793

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Eghi of BMBF Project

Conference

UbiComp/ISWC '22

Sponsor:

UbiComp/ISWC '22: The 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 11 - 15, 2022

Cambridge, United Kingdom

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)7

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Adam GTsangrassoulis A(2024)Design and Development of a Low-Cost Educational Platform for Investigating Human-Centric Lighting (HCL) SettingsComputers10.3390/computers1312033813:12(338)Online publication date: 14-Dec-2024
https://doi.org/10.3390/computers13120338
Sabbella HWeerakoon DGulati MMisra A(2024)PA2BLO: Low-Power, Personalized Audio Badge2024 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom59722.2024.10494427(154-163)Online publication date: 11-Mar-2024
https://doi.org/10.1109/PerCom59722.2024.10494427
Liu MSuh SVargas JZhou BGrünerbl ALukowicz P(2024)A Wearable Multi-modal Edge-Computing System for Real-Time Kitchen Activity RecognitionHuman Activity Recognition and Anomaly Detection10.1007/978-981-97-9003-6_9(132-145)Online publication date: 17-Nov-2024
https://doi.org/10.1007/978-981-97-9003-6_9
Liu MZhou BZhao ZHong HKim HSuh SRey VLukowicz P(2023)FieldHAR: A Fully Integrated End-to-End RTL Framework for Human Activity Recognition with Neural Networks from Heterogeneous Sensors2023 IEEE 34th International Conference on Application-specific Systems, Architectures and Processors (ASAP)10.1109/ASAP57973.2023.00029(110-118)Online publication date: Jul-2023
https://doi.org/10.1109/ASAP57973.2023.00029

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents