More Web Proxy on the site http://driver.im/

research-article

Semantic segmentation-based system for fall detection and post-fall posture classification

Authors:

Mohammed Boulmalf,

Mounir GhoghoAuthors Info & Claims

Volume 117, Issue PB

https://doi.org/10.1016/j.engappai.2022.105616

Published: 01 January 2023 Publication History

Abstract

Fall is one of the most critical issues faced by elders in their daily life. The consequences of falls range from fatal injuries, severe injuries to no injuries. Therefore, an effective system for detecting falls and treating post-fall injuries is important. Unlike wearable sensors, camera-based systems seem more comfortable and flexible for daily life monitoring. However, monitoring human activities in real settings using cameras is not only challenging but also pose privacy issues. To mitigate this problem, we propose a surveillance camera-based framework for fall detection and post-fall classification where the human silhouette is extracted and used instead of raw images. The human silhouette is obtained using a pixel-level classification based on a Multi-Scale Skip Connection Segmentation Network (MSSkip), which is shown to achieve state-of-the-art performance on the validation set from PASCAL VOC 2012 dataset for the class Person with an IoU of 90%. The temporal and spatial variations of the human poses are fed to a Convolutional Long Short Term Memory (ConvLSTM) network to detect whether or not a fall has occurred. The proposed fall detection method achieved an F1-score of 97.68% on the UP Fall dataset, and state-of-art performance on the UR Fall detection database. For post-fall posture classification, the Xception network is shown to achieve an F1-score of 97.85% on our customized post-fall dataset.

References

[1]

Abadi M., Agarwal A., Barham P., Brevdo E., Chen Z., Citro C., Corrado G.S., Davis A., Dean J., Devin M., Ghemawat S., Goodfellow I., Harp A., Irving G., Isard M., Jia Y., Jozefowicz R., Kaiser L., Kudlur M., Levenberg J., Mané D., Monga R., Moore S., Murray D., Olah C., Schuster M., Shlens J., Steiner B., Sutskever I., Talwar K., Tucker P., Vanhoucke V., Vasudevan V., Viégas F., Vinyals O., Warden P., Wattenberg M., Wicke M., Yu Y., Zheng X., TensorFlow: Large-scale machine learning on heterogeneous systems, 2015, [Online]. Available: https://www.tensorflow.org/. Software available from tensorflow.org.

[2]

Ageing and health, 2021, //www.who.int/news-room/fact-sheets/detail/ageing-and-health.

[3]

Ahn, J., Kwak, S., 2018. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4981–4990.

[4]

Al-Sanjary O.I., Ahmed A.A., Jaharadak A.A.B., Ali M.A., Zangana H.M., Detection clone an object movement using an optical flow approach, in: 2018 IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE), IEEE, 2018, pp. 388–394.

[5]

Alexander B.H., Rivara F.P., Wolf M.E., The cost and frequency of hospitalization for fall-related injuries in older adults., Am J Public Health 82 (7) (1992) 1020–1023.

[6]

Badrinarayanan V., Kendall A., Cipolla R., Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. 39 (12) (2017) 2481–2495.

[7]

Barnich O., Van Droogenbroeck M., ViBe: A universal background subtraction algorithm for video sequences, IEEE Trans. Image Process. 20 (6) (2010) 1709–1724.

Digital Library

[8]

Casilari-Perez E., Garcia-Lagos F., A comprehensive study on the use of artificial neural networks in wearable fall detection systems, Expert Syst. Appl. 138 (2019).

[9]

Cho K., Van Merriënboer B., Bahdanau D., Bengio Y., On the properties of neural machine translation: Encoder-decoder approaches, 2014, arXiv preprint arXiv:1409.1259.

[10]

Cho H., Yoon S., Applying singular value decomposition on accelerometer data for 1d convolutional neural network based fall detection, Electron. Lett. 55 (6) (2019) 320–322.

[11]

Chollet, F., 2017. Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1251–1258.

[12]

Cola G., Avvenuti M., Vecchio A., Yang G.-Z., Lo B., An on-node processing approach for anomaly detection in gait, IEEE Sens. J. 15 (11) (2015) 6640–6649.

[13]

Deng J., Dong W., Socher R., Li L.-J., Li K., Fei-Fei L., ImageNet: A Large-Scale Hierarchical Image Database, in: CVPR09, 2009.

[14]

Espinosa R., Ponce H., Gutiérrez S., Martı́nez-Villaseñor L., Brieva J., Moya-Albor E., A vision-based approach for fall detection using multiple cameras and convolutional neural networks: A case study using the UP-fall detection dataset, Comput. Biol. Med. 115 (2019).

[15]

Everingham M., Eslami S.M.A., Van Gool L., Williams C.K.I., Winn J., Zisserman A., The pascal visual object classes challenge: A retrospective, Int. J. Comput. Vis. 111 (1) (2015) 98–136.

Digital Library

[16]

Feng W., Liu R., Zhu M., Fall detection for elderly person care in a vision-based home surveillance environment using a monocular camera, Signal Image Video Process. 8 (6) (2014) 1129–1138.

[17]

Galvão Y.M., Ferreira J., Albuquerque V.A., Barros P., Fernandes B.J., A multimodal approach using deep learning for fall detection, Expert Syst. Appl. 168 (2021).

[18]

Glorot X., Bengio Y., Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, 2010, pp. 249–256.

[19]

Hall C., Essler S., Dandashi J., Corrigan M., Muñoz-Maldonado Y., Juergens A., Wieters S., Drigalla D., Regner J.L., Impact of frailty and anticoagulation status on readmission and mortality rates following falls in patients over 80, in: Baylor University Medical Center Proceedings, Vol. 32, Taylor & Francis, 2019, pp. 181–186. 2.

[20]

Han Q., Zhao H., Min W., Cui H., Zhou X., Zuo K., Liu R., A two-stream approach to fall detection with MobileVGG, IEEE Access 8 (2020) 17556–17566.

[21]

He, K., Zhang, X., Ren, S., Sun, J., 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1026–1034.

[22]

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778.

[23]

Hochreiter S., Schmidhuber J., Long short-term memory, Neural Comput. 9 (8) (1997) 1735–1780.

Digital Library

[24]

Howard A.G., Zhu M., Chen B., Kalenichenko D., Wang W., Weyand T., Andreetto M., Adam H., Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017, arXiv preprint arXiv:1704.04861.

[25]

Hu J., Li L., Lin Y., Wu F., Zhao J., A comparison and strategy of semantic segmentation on remote sensing images, in: The International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery, Springer, 2019, pp. 21–29.

[26]

Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4700–4708.

[27]

Ioffe S., Szegedy C., Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: International Conference on Machine Learning, PMLR, 2015, pp. 448–456.

[28]

Ke, T.-W., Hwang, J.-J., Liu, Z., Yu, S.X., 2018. Adaptive affinity fields for semantic segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 587–602.

[29]

Knowles B., Hanson V.L., The wisdom of older technology (non) users, Commun. ACM 61 (3) (2018) 72–77.

[30]

Kolesnikov A., Lampert C.H., Seed, expand and constrain: Three principles for weakly-supervised image segmentation, in: European Conference on Computer Vision, Springer, 2016, pp. 695–711.

[31]

Kong, S., Fowlkes, C.C., 2018. Recurrent pixel embedding for instance grouping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 9018–9028.

[32]

Kong Y., Huang J., Huang S., Wei Z., Wang S., Learning spatiotemporal representations for human fall detection in surveillance video, J. Vis. Commun. Image Represent. 59 (2019) 215–230.

Digital Library

[33]

Kwolek B., Kepski M., Human fall detection on embedded platform using depth maps and wireless accelerometer, Comput. Methods Programs Biomed. 117 (3) (2014) 489–501.

Digital Library

[34]

Lee J.-S., Tseng H.-H., Development of an enhanced threshold-based fall detection system using smartphones with built-in accelerometers, IEEE Sens. J. 19 (18) (2019) 8293–8302.

[35]

Leite G.V., da Silva G.P., Pedrini H., Three-stream convolutional neural network for human fall detection, in: Deep Learning Applications, Volume 2, Springer, 2021, pp. 49–80.

[36]

Li H., Shrestha A., Heidari H., Le Kernec J., Fioranelli F., Bi-LSTM network for multimodal continuous human activity recognition and fall detection, IEEE Sens. J. 20 (3) (2019) 1191–1201.

[37]

Lin T.-Y., Maire M., Belongie S., Hays J., Perona P., Ramanan D., Dollár P., Zitnick C.L., Microsoft coco: Common objects in context, in: European Conference on Computer Vision, Springer, 2014, pp. 740–755.

[38]

Long, J., Shelhamer, E., Darrell, T., 2015. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3431–3440.

[39]

Maggiori E., Tarabalka Y., Charpiat G., Alliez P., Can semantic labeling methods generalize to any city? The inria aerial image labeling benchmark, in: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE, 2017.

[40]

Mangram A., Dzandu J., Harootunian G., Zhou N., Sohn J., Corneille M., Johnson W., Why elderly patients with ground level falls die within 30 days and beyond, J. Gerontol. Geriatr. Res. 5 (2) (2016) 1–7.

[41]

Martínez-Villaseñor L., Ponce H., Brieva J., Moya-Albor E., Núñez-Martínez J., Peñafort-Asturiano C., UP-fall detection dataset: A multimodal approach, Sensors 19 (9) (2019) 1988.

[42]

Mastorakis G., Ellis T., Makris D., Fall detection without people: A simulation approach tackling video data scarcity, Expert Syst. Appl. 112 (2018) 125–137.

[43]

Mauldin T.R., Canby M.E., Metsis V., Ngu A.H., Rivera C.C., SmartFall: A smartwatch-based fall detection system using deep learning, Sensors 18 (10) (2018) 3363.

[44]

Mirmahboub B., Samavi S., Karimi N., Shirani S., Automatic monocular system for human fall detection based on variations in silhouette area, IEEE Trans. Biomed. Eng. 60 (2) (2012) 427–436.

[45]

Musci M., De Martini D., Blago N., Facchinetti T., Piastra M., Online fall detection using recurrent neural networks on smart wearable devices, IEEE Trans. Emerg. Top. Comput. (2020).

[46]

Nahiduzzaman M., Tasnim M., Newaz N.T., Kaiser M.S., Mahmud M., Machine learning based early fall detection for elderly people with neurological disorder using multimodal data fusion, in: International Conference on Brain Informatics, Springer, 2020, pp. 204–214.

[47]

Nizam Y., Jamil M., Classification of daily life activities for human fall detection: a systematic review of the techniques and approaches, Chall. Trends Multimodal Fall Detect. Healthc. (2020) 137–179.

[48]

Noh, H., Hong, S., Han, B., 2015. Learning deconvolution network for semantic segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1520–1528.

[49]

Núñez-Marcos A., Azkune G., Arganda-Carreras I., Vision-based fall detection with convolutional neural networks, Wirel. Commun. Mobile Comput. 2017 (2017).

[50]

Organization W.H., Ageing W.H.O., Unit L.C., WHO Global Report on Falls Prevention in Older Age, World Health Organization, 2008.

[51]

Pathak, D., Krahenbuhl, P., Darrell, T., 2015. Constrained convolutional neural networks for weakly supervised segmentation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1796–1804.

[52]

Pinheiro, P.O., Collobert, R., 2015. From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1713–1721.

[53]

Radmanesh E., Delrobaei M., Habachi O., Chamani S., Pousset Y., Meghdadi V., A wearable IoT-based fall detection system using triaxial accelerometer and barometric pressure sensor, in: International Symposium on Ubiquitous Networking, Springer, 2019, pp. 158–170.

[54]

Rescio G., Leone A., Siciliano P., Supervised machine learning scheme for electromyography-based pre-fall detection system, Expert Syst. Appl. 100 (2018) 95–105.

[55]

Ronneberger O., Fischer P., Brox T., U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241.

[56]

Rucco R., Sorriso A., Liparoti M., Ferraioli G., Sorrentino P., Ambrosanio M., Baselice F., Type and location of wearable sensors for monitoring falls during static and dynamic tasks in healthy elderly: a review, Sensors 18 (5) (2018) 1613.

[57]

Rumelhart D.E., Hinton G.E., Williams R.J., Learning internal representations by error propagation, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.

[58]

Schuster M., Paliwal K.K., Bidirectional recurrent neural networks, IEEE Trans. Signal Process. 45 (11) (1997) 2673–2681.

Digital Library

[59]

Shi X., Chen Z., Wang H., Yeung D.Y., Wong W.K., Woo W.C., Convolutional LSTM network: A machine learning approach for precipitation nowcasting, Adv. Neural Inf. Process. Syst. 2015 (2015) 802–810.

[60]

Shimoda, W., Yanai, K., 2019. Self-supervised difference detection for weakly-supervised semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5208–5217.

[61]

Shu F., Shu J., An eight-camera fall detection system using human fall pattern recognition via machine learning by a low-cost android box, Sci. Rep. 11 (1) (2021) 1–17.

[62]

Simonyan K., Zisserman A., Very deep convolutional networks for large-scale image recognition, 2014, arXiv preprint arXiv:1409.1556.

[63]

Sterling D.A., O’connor J.A., Bonadies J., Geriatric falls: injury severity is high and disproportionate to mechanism, J. Trauma Acute Care Surg. 50 (1) (2001) 116–119.

[64]

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2818–2826.

[65]

Torti E., Fontanella A., Musci M., Blago N., Pau D., Leporati F., Piastra M., Embedding recurrent neural networks in wearable systems for real-time fall detection, Microprocess. Microsyst. 71 (2019).

Digital Library

[66]

Vallabh P., Malekian R., Fall detection monitoring systems: a comprehensive review, J. Ambient Intell. Humaniz. Comput. 9 (6) (2018) 1809–1833.

[67]

Van Schooten K.S., Pijnappels M., Rispens S.M., Elders P.J., Lips P., Daffertshofer A., Beek P.J., Van Dieen J.H., Daily-life gait quality as predictor of falls in older people: a 1-year prospective cohort study, PLoS One 11 (7) (2016).

[68]

Vincenzo J.L., Glenn J.M., Gray S.M., Gray M., Balance measured by the sway balance smart-device application does not discriminate between older persons with and without a fall history, Aging Clin. Exp. Res. 28 (4) (2016) 679–686.

[69]

Wang, L., Qiao, Y., Tang, X., 2015. Action recognition with trajectory-pooled deep-convolutional descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4305–4314.

[70]

Xu B., Wang N., Chen T., Li M., Empirical evaluation of rectified activations in convolutional network, 2015, arXiv preprint arXiv:1505.00853.

[71]

Zhang Z., Ma X., Wu H., Li Y., Fall detection in videos with trajectory-weighted deep-convolutional rank-pooling descriptor, IEEE Access 7 (2018) 4135–4144.

[72]

Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J., 2017. Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2881–2890.

Cited By

Index Terms

Semantic segmentation-based system for fall detection and post-fall posture classification
1. Applied computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Elderly Assistance Using Wearable Sensors by Detecting Fall and Recognizing Fall Patterns
UbiComp '18: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers

Falling is a serious threat to the elderly people. One severe fall can cause hazardous problems like bone fracture or may lead to some permanent disability or even death. Thus, it has become the need of the hour to continuously monitor the activities of ...
A smartphone-based fall detection system

Falls are a major cause of injuries and hospital admissions among elderly people. Thus, the caregiving process and the quality of life of older adults can be improved by adopting systems for the automatic detection of falls. This paper presents a ...
IoT based fall detection and ambient assisted system for the elderly
Abstract
Falls are considered as risky for the elderly people because it may affect the health of the people. So, in the recent years many elderly fall detection methods has been developed. In the present years many fall detection method had been developed ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Engineering Applications of Artificial Intelligence

Engineering Applications of Artificial Intelligence Volume 117, Issue PB

Jan 2023

378 pages

ISSN:0952-1976

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 January 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents