More Web Proxy on the site http://driver.im/

article

Indian sign language recognition using graph matching on 3D motion captured signs

Authors:

E. Kiran KumarAuthors Info & Claims

Multimedia Tools and Applications, Volume 77, Issue 24

Pages 32063 - 32091

https://doi.org/10.1007/s11042-018-6199-7

Published: 01 December 2018 Publication History

Abstract

A machine cannot easily understand and interpret three-dimensional (3D) data. In this study, we propose the use of graph matching (GM) to enable 3D motion capture for Indian sign language recognition. The sign classification and recognition problem for interpreting 3D motion signs is considered an adaptive GM (AGM) problem. However, the current models for solving an AGM problem have two major drawbacks. First, spatial matching can be performed on a fixed set of frames with a fixed number of nodes. Second, temporal matching divides the entire 3D dataset into a fixed number of pyramids. The proposed approach solves these problems by employing interframe GM for performing spatial matching and employing multiple intraframe GM for performing temporal matching. To test the proposed model, a 3D sign language dataset is created that involves 200 continuous sentences in the sign language through a motion capture setup with eight cameras.The method is also validated on 3D motion capture benchmark action dataset HDM05 and CMU. We demonstrated that our approach increases the accuracy of recognizing signs in continuous sentences.

References

[1]

Agarwal A, Thakur MK (2013) Sign language recognition using microsoft kinect. In: 2013 Sixth international conference on contemporary computing (IC3), IEEE.

[2]

Aggarwal J, Xia L (2014) Human activity recognition from 3d data: a review. Pattern Recogn Lett 48:70---80.

[3]

Almeida SGM, Guimarães FG, Ramírez JA (2014) Feature extraction in brazilian sign language recognition based on phonological structure and using RGB-d sensors. Expert Syst Appl 41(16):7259---7271.

Digital Library

[4]

Ansari ZA, Harit G (2016) Nearest neighbour classification of indian sign language gestures using kinect camera. Sadhana 41(2):161---182.

[5]

Barnachon M, Bouakaz S, Boufama B, Guillou E (2014) Ongoing human action recognition with motion capture. Pattern Recogn 47(1):238---247

Digital Library

[6]

Belgacem S, Chatelain C, Paquet T (2017) Gesture sequence recognition with one shot learned CRF/HMM hybrid model. Image Vis Comput 61:12---21.

[7]

Borzeshi EZ, Piccardi M, Xu RYD (2011) A discriminative prototype selection approach for graph embedding in human action recognition. In: 2011 IEEE International conference on computer vision workshops (ICCV workshops), IEEE.

[8]

Cahill-Rowley K, Rose J (2017) Temporal---spatial reach parameters derived from inertial sensors: Comparison to 3d marker-based motion capture. J Biomech 52:11---16.

[9]

Çeliktutan O, Wolf C, Sankur B, Lombardi E (2014) Fast exact hyper-graph matching with dynamic programming for spatio-temporal data. J Math Imaging Vision 51(1):1---21.

Digital Library

[10]

Chai X, Li G, Chen X, Zhou M, Wu G, Li H (2013) Visualcomm: a tool to support communication between deaf and hearing persons with the kinect. In: Proceedings of the 15th international ACM SIGACCESS conference on computers and accessibility, ACM, p 76

Digital Library

[11]

Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996---1002

[12]

Duan J, Zhou S, Wan J, Guo X, Li SZ (2016) Multi-modality fusion based on consensus-voting and 3d convolution for isolated gesture recognition. arXiv:1611.06689

[13]

Geng L, Ma X, Wang H, Gu J, Li Y (2014) Chinese sign language recognition with 3d hand motion trajectories and depth images. In: Proceeding of the 11th world congress on intelligent control and automation, IEEE.

[14]

Grest D, Krüger V Gradient-enhanced particle filter for vision-based motion capture. In: Human motion---understanding, modeling, capture and animation. Springer, Berlin, pp 28---41

Digital Library

[15]

Gärtner T, Flach P, Wrobel S (2003) On graph kernels: hardness results and efficient alternatives. In: Learning theory and kernel machines. Springer, Berlin, pp 129---143

[16]

Guess TM, Razu S, Jahandar A, Skubic M, Huo Z (2017) Comparison of 3d joint angles measured with the kinect 2.0 skeletal tracker versus a marker-based motion capture system. J Appl Biomech 33(2):176---181.

[17]

Han F, Reily B, Hoff W, Zhang H (2017) Space-time representation of people based on 3d skeletal data: a review. Comput Vis Image Underst 158:85---105.

Digital Library

[18]

Huang P, Hilton A, Starck J (2010) Shape similarity for 3d video sequences of people. Int J Comput Vis 89(2-3):362---381

Digital Library

[19]

Jeong YS, Jeong MK, Omitaomu OA (2011) Weighted dynamic time warping for time series classification. Pattern Recogn 44(9):2231---2240

Digital Library

[20]

Junejo IN, Aghbari ZA (2012) Using SAX representation for human action recognition. J Vis Commun Image Represent 23(6):853---861

Digital Library

[21]

Kakadiaris I, Barrón C Model-based human motion capture. In: Handbook of mathematical models in computer vision, Springer, pp 325---340

[22]

Kishore PVV, Kumar DA, Sastry ASCS, Kumar EK (2018) Motionlets matching with adaptive kernels for 3-d indian sign language recognition. IEEE Sensors J 18(8):3327---3337.

[23]

Kumar Eepuri K, Kishore PSSASC, Maddala TKK, Kumar Dande A (2018) Training CNNs, for 3d sign language recognition with color texture coded joint angular displacement maps. IEEE Signal Processing Letters :1---1.

[24]

Kumar P, Gauba H, Roy PP, Dogra DP (2017) Coupled hmm-based multi-sensor data fusion for sign language recognition. Pattern Recogn Lett 86:1---8

[25]

Kumar P, Gauba H, Roy PP, Dogra DP (2017) A multimodal framework for sensor based sign language recognition. Neurocomputing 259:21---38

[26]

Kushwah MS, Sharma M, Jain K, Chopra A (2016) Sign language interpretation using pseudo glove. In: Proceeding of international conference on intelligent communication, control and devices, Springer, Singapore, pp 9---18

[27]

Leightley D, Li B, McPhee JS, Yap MH, Darby J (2014) Exemplar-based human action recognition with template matching from a stream of motion capture. In: Lecture notes in computer science, Springer international publishing, pp 12---20

[28]

Li K, Zhou Z, Lee CH (2016) Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. ACM Transactions on Accessible Computing 8(2):1---23.

Digital Library

[29]

Li M, Leung H, Liu Z, Zhou L (2016) 3d human motion retrieval using graph kernels based on adaptive graph construction. Comput Graph 54:104---112

Digital Library

[30]

Li SZ, Yu B, Wu W, Su SZ, Ji RR (2015) Feature learning based on SAE---PCA network for human gesture recognition in RGBD images. Neurocomputing 151:565---573.

[31]

Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In: AAAI, vol 30, pp 1266---1272

Digital Library

[32]

Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: recognizing complex activities from sensor data. In: IJCAI, vol 2015, pp 1617---1623

Digital Library

[33]

Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108---115

Digital Library

[34]

Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimedia Tools and Applications 76(8):10701---10719

Digital Library

[35]

Mapari RB, Kharat G (2016) American static signs recognition using leap motion sensor. In: Proceedings of the Second international conference on information and communication technology for competitive strategies, ACM, p 67

Digital Library

[36]

Moeslund TB, Granum E (2001) A survey of computer vision-based human motion capture. Comput Vis Image Underst 81(3):231---268

Digital Library

[37]

Nai W, Liu Y, Rempel D, Wang Y (2017) Fast hand posture classification using depth features extracted from random line segments. Pattern Recogn 65:1---10.

Digital Library

[38]

Park S, Park H, Kim J, Adeli H (2015) 3d displacement measurement model for health monitoring of structures using a motion capture system. Measurement 59:352---362.

[39]

Rao GA, Kishore P (2017) Selfie video based continuous indian sign language recognition system. Ain Shams Engineering Journal.

[40]

Rucco R, Agosti V, Jacini F, Sorrentino P, Varriale P, Stefano MD, Milan G, Montella P, Sorrentino G (2017) Spatio-temporal and kinematic gait analysis in patients with frontotemporal dementia and alzheimer's disease through 3d motion capture. Gait Posture 52:312---317.

[41]

Sandler W (2017) The challenge of sign language phonology. Annual Review of Linguistics 3:43---63

[42]

Sun C, Zhang T, Bao BK, Xu C, Mei T (2013) Discriminative exemplar coding for sign language recognition with kinect. IEEE Transactions on Cybernetics 43(5):1418---1428.

[43]

Sun S, Luo C, Chen J (2017) A review of natural language processing techniques for opinion mining systems. Information Fusion 36:10---25.

Digital Library

[44]

Sun Y, Bray M, Thayananthan A, Yuan B, Torr P (2006) Regression-based human motion capture from voxel data. In: Procedings of the british machine vision conference 2006. British machine vision association

[45]

Ta AP, Wolf C, Lavoue G, Baskurt A (2010) Recognizing and localizing individual activities through graph matching. In: 2010 7Th IEEE international conference on advanced video and signal based surveillance, IEEE.

Digital Library

[46]

Xiao Q, Wang Y, Wang H (2014) Motion retrieval using weighted graph matching. Soft Comput 19(1):133---144.

Digital Library

[47]

Yang C, Cheung G, Stankovic V (2017) Estimating heart rate and rhythm via 3d motion tracking in depth video. IEEE Trans Multimedia 19(7):1625---1636.

Digital Library

[48]

Zhang W, Liu Z, Zhou L, Leung H, Chan AB (2017) Martial arts, dancing and sports dataset: a challenging stereo and multi-view dataset for 3d human pose estimation. Image Vis Comput 61:22---39.

Digital Library

[49]

Zhang Z, Kurakin AV Dynamic hand gesture recognition using depth data (2017). US Patent 9,536, 135

Cited By

Alam IHameed AZiar R(2024)Exploring Sign Language Detection on SmartphonesAdvances in Human-Computer Interaction10.1155/2024/14875002024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/1487500
Alyami SLuqman HHammoudeh M(2024)Reviewing 25 years of continuous sign language recognition researchInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10377461:5Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103774
Kumar DSastry AKishore PKumar E(2022)3D sign language recognition using spatio temporal graph kernelsJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2018.11.00834:2(143-152)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2018.11.008
Show More Cited By

Indian sign language recognition using graph matching on 3D motion captured signs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks

Recommendations

Early estimation model for 3D-discrete indian sign language recognition using graph matching
Abstract
Machine translation of sign language is a critical task of computer vision. In this work, we propose to use 3D motion capture technology for sign capture and graph matching for sign recognition. Two problems related to 3D sign matching ...
3D sign language recognition using spatio temporal graph kernels
Abstract
3D Sign language recognition is challenging from capturing to recognition. 3D signs are a set of spatio temporal variations of hands and fingers with respect to face, head and torso. 3D motion capture technology has enabled us to ...
SportsCap: Monocular 3D Human Motion Capture and Fine-Grained Understanding in Challenging Sports Videos
Abstract
Markerless motion capture and understanding of professional non-daily human movements is an important yet unsolved task, which suffers from complex motion patterns and severe self-occlusion, especially for the monocular setting. In this paper, we ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Multimedia Tools and Applications

Multimedia Tools and Applications Volume 77, Issue 24

December 2018

761 pages

ISSN:1380-7501

Issue’s Table of Contents

Copyright © Copyright © 2018 Springer Science+Business Media, LLC, part of Springer Nature.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 December 2018

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alam IHameed AZiar R(2024)Exploring Sign Language Detection on SmartphonesAdvances in Human-Computer Interaction10.1155/2024/14875002024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/1487500
Alyami SLuqman HHammoudeh M(2024)Reviewing 25 years of continuous sign language recognition researchInformation Processing and Management: an International Journal10.1016/j.ipm.2024.10377461:5Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.ipm.2024.103774
Kumar DSastry AKishore PKumar E(2022)3D sign language recognition using spatio temporal graph kernelsJournal of King Saud University - Computer and Information Sciences10.1016/j.jksuci.2018.11.00834:2(143-152)Online publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.jksuci.2018.11.008
Patel DJoshi J(2022)Deep Leaning Based Static Indian-Gujarati Sign Language Gesture RecognitionSN Computer Science10.1007/s42979-022-01254-23:5Online publication date: 16-Jul-2022
https://dl.acm.org/doi/10.1007/s42979-022-01254-2
Sahana TBasu SNasipuri MMollah A(2022)MRCS: multi-radii circular signature based feature descriptor for hand gesture recognitionMultimedia Tools and Applications10.1007/s11042-021-11743-w81:6(8539-8560)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s11042-021-11743-w
Maddala TKishore PEepuri KDande A(2019)YogaNet: 3-D Yoga Asana Recognition Using Joint Angular Displacement Maps With ConvNetsIEEE Transactions on Multimedia10.1109/TMM.2019.290488021:10(2492-2503)Online publication date: 23-Sep-2019
https://dl.acm.org/doi/10.1109/TMM.2019.2904880

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents