[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

SenseMLP: a parallel MLP architecture for sensor-based human activity recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Human activity recognition (HAR) with wearable inertial sensors is a burgeoning field, propelled by advances in sensor technology. Deep learning methods for HAR have notably enhanced recognition accuracy in recent years. Nonetheless, the complexity of previous models often impedes their use in real-life scenarios, particularly in online applications. Addressing this gap, we introduce SenseMLP, a novel approach employing a multi-layer perceptron (MLP) neural network architecture. SenseMLP features three parallel MLP branches that independently process and integrate features across the time, channel, and frequency dimensions. This structure not only simplifies the model but also significantly reduces the number of required parameters compared to previous deep learning HAR frameworks. We conducted comprehensive evaluations of SenseMLP against benchmark HAR datasets, including PAMAP2, OPPORTUNITY, USC-HAD, and SKODA. Our findings demonstrate that SenseMLP not only achieves state-of-the-art performance in terms of accuracy but also boasts fewer parameters and lower floating-point operations per second. For further research and application in the field, the source code of SenseMLP is available at https://github.com/forfrees/SenseMLP.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

No datasets were generated or analysed during the current study.

References

  1. Kim, E., Helal, S., Cook, D.: Human activity recognition and pattern discovery. IEEE Pervasive Comput. 9(1), 48–53 (2009)

    Article  Google Scholar 

  2. Rashidi, P., Cook, D.J., Holder, L.B., Schmitter-Edgecombe, M.: Discovering activities to recognize and track in a smart environment. IEEE Trans. Knowl. Data Eng. 23(4), 527–539 (2010)

    Article  Google Scholar 

  3. Acampora, G., Cook, D.J., Rashidi, P., Vasilakos, A.V.: A survey on ambient intelligence in healthcare. Proc. IEEE 101(12), 2470–2494 (2013)

    Article  Google Scholar 

  4. Zhou, X., Liang, W., Kevin, I., Wang, K., Wang, H., Yang, L.T., Jin, Q.: Deep-learning-enhanced human activity recognition for internet of healthcare things. IEEE Internet Things J. 7(7), 6429–6438 (2020)

    Article  Google Scholar 

  5. Cook, D.J., Augusto, J.C., Jakkula, V.R.: Ambient intelligence: technologies, applications, and opportunities. Pervasive Mob. Comput. 5(4), 277–298 (2009)

    Article  Google Scholar 

  6. Chen, L., Nugent, C.D., Wang, H.: A knowledge-driven approach to activity recognition in smart homes. IEEE Trans. Knowl. Data Eng. 24(6), 961–974 (2011)

    Article  Google Scholar 

  7. Chen, L., Hoey, J., Nugent, C.D., Cook, D.J., Yu, Z.: Sensor-based activity recognition. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(6), 790–808 (2012)

    Article  Google Scholar 

  8. Lara, O.D., Labrador, M.A.: A survey on human activity recognition using wearable sensors. IEEE Commun. Surv. Tutor. 15(3), 1192–1209 (2012)

    Article  Google Scholar 

  9. Zeng, M., Nguyen, L.T., Yu, B., Mengshoel, O.J., Zhu, J., Wu, P., Zhang, J.: Convolutional neural networks for human activity recognition using mobile sensors. In: 6th International Conference on Mobile Computing, Applications and Services, pp. 197–205. IEEE, Austin, TX, USA (2014)

  10. Jiang, W., Yin, Z.: Human activity recognition using wearable sensors by deep convolutional neural networks. In: 23rd ACM International Conference on Multimedia, pp. 1307–1310. ACM, New York, NY, USA (2015)

  11. Ha, S., Yun, J.-M., Choi, S.: Multi-modal convolutional neural networks for activity recognition. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 3017–3022. IEEE, Hong Kong, China (2015)

  12. Ronao, C.A., Cho, S.-B.: Human activity recognition with smartphone sensors using deep learning neural networks. Expert Syst. Appl. 59, 235–244 (2016)

    Article  Google Scholar 

  13. Bianchi, V., Bassoli, M., Lombardo, G., Fornacciari, P., Mordonini, M., De Munari, I.: IoT wearable sensor and deep learning: an integrated approach for personalized human activity recognition in a smart home environment. IEEE Internet Things J. 6(5), 8553–8562 (2019)

    Article  Google Scholar 

  14. Gil-Martín, M., San-Segundo, R., Fernandez-Martinez, F., Ferreiros-López, J.: Improving physical activity recognition using a new deep learning architecture and post-processing techniques. Eng. Appl. Artif. Intell. 92, 103679 (2020)

    Article  Google Scholar 

  15. Pham, C., Nguyen-Thai, S., Tran-Quang, H., Tran, S., Vu, H., Tran, T.-H., Le, T.-L.: SensCapsNet: deep neural network for non-obtrusive sensing based human activity recognition. IEEE Access 8, 86934–86946 (2020)

    Article  Google Scholar 

  16. Cruciani, F., Vafeiadis, A., Nugent, C., Cleland, I., McCullagh, P., Votis, K., Giakoumis, D., Tzovaras, D., Chen, L., Hamzaoui, R.: Feature learning for human activity recognition using convolutional neural networks: a case study for inertial measurement unit and audio data. CCF Trans. Pervasive Comput. Interact. 2(1), 18–32 (2020)

    Article  Google Scholar 

  17. Ordóñez, F.J., Roggen, D.: Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1), 115 (2016)

    Article  Google Scholar 

  18. Yao, S., Hu, S., Zhao, Y., Zhang, A., Abdelzaher, T.: DeepSense: a unified deep learning framework for time-series mobile sensing data processing. In: Proceedings of the 26th International Conference on World Wide Web, pp. 351–360. Perth, Australia (2017)

  19. Xu, C., Chai, D., He, J., Zhang, X., Duan, S.: InnoHAR: a deep neural network for complex human activity recognition. IEEE Access 7, 9893–9902 (2019)

    Article  Google Scholar 

  20. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9. IEEE, Boston, MA, USA (2015)

  21. Xia, K., Huang, J., Wang, H.: LSTM-CNN architecture for human activity recognition. IEEE Access 8, 56855–56866 (2020)

    Article  Google Scholar 

  22. Murahari, V.S., Plötz, T.: On attention models for human activity recognition. In: Proceedings of the 2018 ACM International Symposium on Wearable Computers, pp. 100–103. ACM, New York, NY, USA (2018)

  23. Zeng, M., Gao, H., Yu, T., Mengshoel, O.J., Langseth, H., Lane, I., Liu, X.: Understanding and improving recurrent networks for human activity recognition by continuous attention. In: Proceedings of the 2018 ACM International Symposium on Wearable Computers, pp. 56–63. ACM, New York, NY, USA (2018)

  24. Ma, H., Li, W., Zhang, X., Gao, S., Lu, S.: AttnSense: multi-level attention mechanism for multimodal human activity recognition. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3109–3115. AAAI Press, Macao, China (2019)

  25. Gao, W., Zhang, L., Teng, Q., He, J., Wu, H.: DanHAR: dual attention network for multimodal human activity recognition using wearable sensors. Appl. Soft Comput. 111, 107728 (2021)

    Article  Google Scholar 

  26. Khan, Z.N., Ahmad, J.: Attention induced multi-head convolutional neural network for human activity recognition. Appl. Soft Comput. 110, 107671 (2021)

    Article  Google Scholar 

  27. Gao, W., Zhang, L., Huang, W., Min, F., He, J., Song, A.: Deep neural networks for sensor-based human activity recognition using selective kernel convolution. IEEE Trans. Instrum. Meas. 70, 1–13 (2021)

    Google Scholar 

  28. Tang, Y., Zhang, L., Teng, Q., Min, F., Song, A.: Triple cross-domain attention on human activity recognition using wearable sensors. IEEE Trans. Emerg. Top. Comput. Intell. 6(5), 1167–1176 (2022)

    Article  Google Scholar 

  29. Yao, S., Zhao, Y., Shao, H., Liu, D., Liu, S., Hao, Y., Piao, A., Hu, S., Lu, S., Abdelzaher, T.F.: SADeepSense: self-attention deep learning framework for heterogeneous on-device sensors in Internet of Things applications. In: IEEE INFOCOM 2019—IEEE Conference on Computer Communications, pp. 1243–1251 (2019). https://doi.org/10.1109/INFOCOM.2019.8737500

  30. Singh, S.P., Sharma, M.K., Lay-Ekuakille, A., Gangwar, D., Gupta, S.: Deep ConvLSTM with self-attention for human activity decoding using wearable sensors. IEEE Sens. J. 21(6), 8575–8582 (2021). https://doi.org/10.1109/JSEN.2020.3045135

    Article  Google Scholar 

  31. Mahmud, S., Tonmoy, M.T.H., Bhaumik, K.K., Rahman, A.M., Amin, M.A., Shoyaib, M., Khan, M.A.H., Ali, A.: Human activity recognition from wearable sensor data using self-attention. In: proceedings of 24th European Conference on Artificial Intelligence, 29 August–8 September 2020, Santiago de Compostela, Spain (2020)

  32. Dirgová Luptáková, I., Kubovcík, M., Pospíchal, J.: Wearable sensor-based human activity recognition with transformer model. Sensors (2022). https://doi.org/10.3390/s22051911

    Article  Google Scholar 

  33. Kim, Y.-W., Cho, W.-H., Kim, K.-S., Lee, S.: Inertial-measurement-unit-based novel human activity recognition algorithm using conformer. Sensors (2022). https://doi.org/10.3390/s22103932

    Article  Google Scholar 

  34. Zhang, Y., Wang, L., Chen, H., Tian, A., Zhou, S., Guo, Y.: IF-ConvTransformer: a framework for human activity recognition using IMU fusion and ConvTransformer. Proc. ACM Interact. Mobile Wearable. Ubiquit. Technol. 6(2), 1–26 (2022)

    Google Scholar 

  35. Essa, E., Abdelmaksoud, I.R.: Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors. Knowl. Based Syst. 278, 110867 (2023). https://doi.org/10.1016/j.knosys.2023.110867

    Article  Google Scholar 

  36. Tolstikhin, I.O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J.: MLP-mixer: an all-MLP architecture for vision. Adv. Neural Inf. Process. Syst. 34, 24261–24272 (2021)

    Google Scholar 

  37. Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: 2012 16th International Symposium on Wearable Computers, pp. 108–109. IEEE, Newcastle, UK (2012)

  38. Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Förster, K., Tröster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A.: Collecting complex activity datasets in highly rich networked sensor environments. In: 2010 Seventh International Conference on Networked Sensing Systems, pp. 233–240. IEEE, Kassel, Germany (2010)

  39. Zhang, M., Sawchuk, A.A.: USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 1036–1043. ACM, New York, NY, USA (2012)

  40. Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tröster, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Comput. 7(2), 42–50 (2008)

    Article  Google Scholar 

  41. Tong, L., Ma, H., Lin, Q., He, J., Peng, L.: A novel deep learning Bi-GRU-I model for real-time human activity recognition using inertial sensors. IEEE Sens. J. 22(6), 6164–6174 (2022)

    Article  Google Scholar 

  42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010. Long Beach, CA, USA (2017)

  43. Anmol, G., James, Q., Chung-Cheng, C., Niki, P., Yu, Z., Jiahui, Y., Wei, H., Shibo, W., Zhengdong, Z., Yonghui, W., Ruoming, P.: Conformer: convolution-augmented transformer for speech recognition. INTERSPEECH, Shanghai, China (2020)

  44. Xiao, S., Wang, S., Huang, Z., Wang, Y., Jiang, H.: Two-stream transformer network for sensor-based human activity recognition. Neurocomputing 512, 253–268 (2022). https://doi.org/10.1016/j.neucom.2022.09.099

    Article  Google Scholar 

  45. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics, NAACL, Minneapolis, MN, USA (2019)

  46. Huang, C.-Z.A., Vaswani, A., Uszkoreit, J., Shazeer, N.M., Simon, I., Hawthorne, C., Dai, A.M., Hoffman, M.D., Dinculescu, M., Eck, D.: Music transformer: generating music with long-term structure. In: International Conference on Learning Representations, Vancouver, Canada (2018)

  47. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: transformers for image recognition at scale, In: Proceedings of the 9th International Conference on Learning Representations, Virtual Event, Austria (2021)

  48. Hendrycks, D., Gimpel, K.: Gaussian error linear units (GELUs) (2016). arXiv preprint. http://arxiv.org/abs/1606.08415

  49. Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., Tay, Y.: The efficiency misnomer (2021). In: The Tenth International Conference on Learning Representations, Virtual Event (2022)

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, China

    Weilin Li, Jiaming Guo & Hong Wu

Authors
  1. Weilin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaming Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.W. developed the idea. W.L. implemented the method. W.L. and J.G did the experiments. H.W. wrote and revised the main manuscript text, and W.L wrote the experiment section. J.G. prepared Fig. 1. All authors reviewed the manuscript.

Corresponding author

Correspondence to Hong Wu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Communicated by Bing-kun Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, W., Guo, J. & Wu, H. SenseMLP: a parallel MLP architecture for sensor-based human activity recognition. Multimedia Systems 30, 182 (2024). https://doi.org/10.1007/s00530-024-01384-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01384-y

Keywords

Search

Navigation