More Web Proxy on the site http://driver.im/

research-article

HAKA: : HierArchical Knowledge Acquisition in a sign language tutor

Authors:

Itsaso Rodríguez-Moreno,

José María Martínez-Otzeta,

Basilio SierraAuthors Info & Claims

Volume 215, Issue C

https://doi.org/10.1016/j.eswa.2022.119365

Published: 01 April 2023 Publication History

Highlights

•

A tutor for learning the Spanish Sign Language basic hand configurations.

•

Hand landmark extraction from a sequence of images.

•

Similarities between configurations using Procrustes analysis.

•

Multidimensional scaling analysis of the clustering of the hand configurations.

Abstract

Communication between people from different communities can sometimes be hampered by the lack of knowledge of each other's language. A large number of people needs to learn a language in order to ensure a fluid communication or want to do it just out of intellectual curiosity. To assist language learners' needs tutor tools have been developed. In this paper we present a tutor for learning the basic 42 hand configurations of the Spanish Sign Language, as well as more than one hundred of common words. This tutor registers the user image from an off-the-shelf webcam and challenges her to perform the hand configuration she chooses to practice. The system looks for the configuration, out of the 42 in its database, closest to the configuration performed by the user, and shows it to her, to help her to improve through knowledge of her errors in real time. The similarities between configurations are computed using Procrustes analysis. A table with the most frequent mistakes is also recorded and available to the user. The user may advance to choose a word and practice the hand configurations needed for that word. Sign languages have been historically neglected and deaf people still face important challenges in their daily activities. This research is a first step in the development of a Spanish Sign Language tutor and the tool is available as open source. A multidimensional scaling analysis of the clustering of the 42 hand configurations induced by Procrustes similarity is also presented.

References

[1]

M.A. Ahmed, B.B. Zaidan, A.A. Zaidan, M.M. Salih, M.M. Lakulu, A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017, Sensors 18 (2018) 2208.

[2]

Aktaş, M., Gökberk, B., & Akarun, L. (2019). Recognizing non-manual signs in Turkish sign language. 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA), (pp. 1–6).

[3]

M. Al-Qurishi, T. Khalid, R. Souissi, Deep learning for sign language recognition: Current techniques, benchmarks, and open issues, IEEE Access, 2021.

[4]

S.G. Azar, H. Seyedarabi, Trajectory-based recognition of dynamic Persian sign language using hidden Markov model, Computer Speech & Language 61 (2020).

[5]

L.E. Baum, T. Petrie, Statistical inference for probabilistic functions of finite state Markov chains, The annals of mathematical statistics 37 (1966) 1554–1563.

[6]

Boháček, M., & Hrúz, M. (2022). Sign Pose-based Transformer for Word-level Sign Language Recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, (pp. 182–191).

[7]

Bradski, G. R. (1998). Real time face and object tracking as a component of a perceptual user interface. Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No. 98EX201), (pp. 214–219).

[8]

X. Chen, D. Zou, F. Su, Twenty-five years of computer-assisted language learning: A topic modeling analysis, Language Learning & Technology 25 (2021) 151–185.

[9]

C. Cortes, V. Vapnik, Support vector machine, Machine Learning 20 (1995) 273–297.

[10]

Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), 1, pp. 886–893.

[11]

De Marco, R. M., & Foulds, R. A. (2003). Data recording and analysis of American Sign Language. 2003 IEEE 29th Annual Proceedings of Bioengineering Conference, (pp. 49–50).

[12]

Dryden, I. L., & Mardia, K. V. (2016). Statistical shape analysis: with applications in R (Vol. 995). John Wiley & Sons.

[13]

Eberhard, D. M., Simons, G. F., & Fennig, C. D. (2022). Ethnologue: Languages of the world. Ethnologue: Languages of the world. Dallas, TX: SIL International.

[14]

U. Fadlilah, B. Handaga, et al., The development of android for Indonesian sign language using tensorflow lite and CNN: An initial study, Journal of Physics: Conference Series 1858 (2021).

[15]

M.B. Fazi, Beyond human: Deep learning, explainability and representation, Theory, Culture & Society 38 (2021) 55–77.

[16]

L. Gao, H. Li, Z. Liu, Z. Liu, L. Wan, W. Feng, RNN-transducer based Chinese sign language recognition, Neurocomputing 434 (2021) 45–54.

[17]

S. Garcia, J. Derrac, J. Cano, F. Herrera, Prototype selection for nearest neighbor classification: Taxonomy and empirical study, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2012) 417–435.

Digital Library

[18]

Golub, G. H., & Van Loan, C. F. (2013). Matrix computations. JHU press.

[19]

J.C. Gower, Generalized procrustes analysis, Psychometrika 40 (1975) 33–51.

[20]

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, (pp. 770–778).

[21]

R. Ibañez, Á. Soria, A. Teyseyre, G. Rodrı́guez, M. Campo, Approximate string matching: A lightweight approach to recognize gestures with Kinect, Pattern Recognition 62 (2017) 73–86.

[22]

Jiang, Y., Bosch, N., Baker, R. S., Paquette, L., Ocumpaugh, J., Andres, J. M., . . . Biswas, G. (2018). Expert feature-engineering vs. deep neural networks: which is better for sensor-free affect detection? International conference on artificial intelligence in education, (pp. 198–211).

[23]

E. Mazereanu, Market size of the global language services industry, 2009–2022, Market size of the global language services industry (2019) 2009–2022.

[24]

Quinn, M., & Olszewska, J. I. (2019). British sign language recognition in the wild based on multi-class SVM. 2019 federated conference on computer science and information systems (FedCSIS), (pp. 81–86).

[25]

R. Rastgoo, K. Kiani, S. Escalera, Sign language recognition: A deep survey, Expert Systems with Applications 164 (2021).

[26]

Révy, G., Hadházi, D., & Hullám, G. (2022). Towards Hand-Over-Face Gesture Detection. 29th Minisymposium of the Department of Measurement and Information Systems, (pp. 58–61).

[27]

Rocha, J., Lensk, J., Ferreira, T., & Ferreira, M. (2020). Towards a tool to translate brazilian sign language (libras) to brazilian portuguese and improve communication with deaf. 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), (pp. 1–4).

[28]

P.P. Roy, P. Kumar, B.-G. Kim, An efficient sign language recognition (SLR) system using Camshift tracker and hidden Markov model (HMM), SN Computer Science 2 (2021) 1–15.

[29]

O. Sevli, N. Kemaloğlu, Turkish sign language digits classification with CNN using different optimizers, International Advanced Researches and Engineering Journal 4 (2020) 200–207.

[30]

A. Sharma, N. Sharma, Y. Saxena, A. Singh, D. Sadhya, Benchmarking deep neural network approaches for Indian Sign Language recognition, Neural Computing and Applications 33 (2021) 6685–6696.

[31]

Taskiran, M., Killioglu, M., & Kahraman, N. (2018). A real-time system for recognition of American sign language by using deep learning. 2018 41st international conference on telecommunications and signal processing (TSP), (pp. 1–5).

[32]

J. Tremblay, K. Singh, A. Fern, E.S. Kirton, S. He, T. Woyke, …., S.G. Tringe, Primer and platform effects on 16S rRNA tag sequencing, Frontiers in microbiology 6 (2015) 771.

[33]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, …., I. Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems 30 (2017).

[34]

R.Y. Wang, J. Popović, Real-time hand-tracking with a color glove, ACM Transactions on Graphics (TOG) 28 (2009) 1–8.

[35]

Zabala, U., Rodriguez, I., Martı́nez-Otzeta, J. M., Irigoien, I., & Lazkano, E. (2021). Quantitative analysis of robot gesticulation behavior. Autonomous Robots, 45, 175–189.

[36]

Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., & Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214.

[37]

R. Zhang, D. Zou, Types, purposes, and effectiveness of state-of-the-art technologies for second and foreign language learning, Computer Assisted Language Learning 35 (2022) 696–742.

Index Terms

HAKA: HierArchical Knowledge Acquisition in a sign language tutor

Index terms have been assigned to the content through auto-classification.

Recommendations

Encouraging Persons with Hearing Problem to Learn Sign Language by Internet Websites
ICALT '08: Proceedings of the 2008 Eighth IEEE International Conference on Advanced Learning Technologies

Nowadays the Internet users are from different ages and groups. Disabled people are a group of the Internet users. Some websites are especially created for these people. One group of the disabled people are deaf persons. They have a special talking ...
Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation

Automated synthesis of American Sign Language (ASL) animations will benefit people who are deaf with low English literacy.Annotated ASL motion-capture corpora enable researchers to produce animations with complex spatial and linguistic phenomena.We ...
Integrating MultiWordNet with Italian Sign Language lexical resources

A novel Italian Sign Language MultiWordNet (LMWN), which integrates the MultiWordNet (MWN) lexical database with the Italian Sign Language (LIS), is presented in this paper. The approach relies on LIS lexical resources which support and help to search ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 215, Issue C

Apr 2023

1634 pages

ISSN:0957-4174

Issue’s Table of Contents

The Author(s).

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 April 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents