Article

A Parallel Multistream Model for Integration of Sign Language Recognition and Lip Motion

Authors:

Jiyong Ma,

Wen Gao,

Rui WangAuthors Info & Claims

ICMI '00: Proceedings of the Third International Conference on Advances in Multimodal Interfaces

Pages 582 - 589

Published: 14 October 2000 Publication History

Abstract

The parallel multistream model is proposed for integration sign language recognition and lip motion. The different time scales existing in sign language and lip motion can be tackled well using this approach. Primary experimental results have shown that this approach is efficient for integration of sign language recognition and lip motion. The promising results indicated that parallel multistream model can be a good solution in the framework of multimodal data fusion. An approach to recognize sign language with scalability with the size of vocabulary and a fast approach to locate lip corners are also proposed in this paper.

References

[1]

Ho-Sub Yoon. Jung Soh. Byung-Woo Min. Hyun Seung Yang. Recognition of alphabetical hand gestures using hidden Markov model. IEICE Transactions on Fundamentals of Electronics Communications & Computer Sciences, vol.E82-A, no.7, July 1999, pp.1358-66. Publisher: Inst. Electron. Inf. & Commun. Eng, Japan.

Google Scholar

[2]

Vogler C. Metaxas D. Parallel hidden Markov models for American Sign Language recognition. Proceedings of the Seventh IEEE International Conference on Computer Vision. IEEE Comput. Soc. Part vol.1, 1999, pp.116-22 vol.1. Los Alamitos, CA, USA.

Google Scholar

[3]

Luettin, J., Thacker, N.A. and Beet, S.W. Speechreading using shape and intensity information, Proc. 4th ICSLP Confference, Philadephia, PA, USA, 1996.

Google Scholar

[4]

Michael Vogt. Fast matching of a dynamic lip model to color video sequence under regular illumination conditions. In speechreading by human and machines, NATO ASI series F., Vol 150, D.G. Stork, M.E. Hennecke (eds), pages 199-407, 1996.

Google Scholar

[5]

Jiyong Ma, Wen Gao, Jiangqin Wu and Chunli Wang, A Continuous Chinese Sign Language recognition system, pp428-433, 28-31 March, FG'2000, Grenoble, France.

Crossref

Google Scholar

[6]

J. Matas. Color-based Object Recognition. Ph.D thesis, University of Surry, 1995.

Google Scholar

[7]

Karin Sobottka and Ioannis Pitas. Face Localization and facial feature extraction based on color shape and color information. ICIP, 1996.

Google Scholar

[8]

Uwe Meier, Rainer Stifelhagen, Jie Yang and Alex Waibel. Towards unrestricted lip reading. ICMI, 1999.

Google Scholar

[9]

Bourlard, H., Non-stationary multi-channel (multi-stream) processing towards robust and adaptive ASR, Proc. Tampere workshop on robust methods for speech recognition in aderse conditions, 1995, pp.1-10.

Google Scholar

[10]

Mirghafori, N., A multi-band approach to automatic speech recognition, PhD dissertation, University of California at Berkely, Dec 1998. Reprinted as ICSI Technical report, ICSI TR-99-04.

Crossref

Google Scholar

Cited By

View all

Kelly DReilly Delannoy JMc Donald JMarkham CCrowley JIvanov YWren CGatica-Perez DJohnston MStiefelhagen R(2009)A framework for continuous multimodal sign language recognitionProceedings of the 2009 international conference on Multimodal interfaces10.1145/1647314.1647387(351-358)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1647314.1647387
Aran OBurger TCaplier AAkarun L(2009)A belief-based sequential fusion approach for fusing manual signs and non-manual signalsPattern Recognition10.1016/j.patcog.2008.09.01042:5(812-822)Online publication date: 1-May-2009
https://dl.acm.org/doi/10.1016/j.patcog.2008.09.010
Caplier AStillittano SAran OAkarun LBailly GBeautemps DAboutabit NBurger T(2007)Image and video for hearing impaired peopleJournal on Image and Video Processing10.1155/2007/456412007:5(1-14)Online publication date: 1-Jan-2007
https://dl.acm.org/doi/10.1155/2007/45641
Show More Cited By

Index Terms

A Parallel Multistream Model for Integration of Sign Language Recognition and Lip Motion
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding

Recommendations

Multimodal speaker/speech recognition using lip motion, lip texture and audio
Special section: Multimodal human-computer interfaces

We present a new multimodal speaker/speech recognition system that integrates audio, lip texture and lip motion modalities. Fusion of audio and face texture modalities has been investigated in the literature before. The emphasis of this work is to ...
Gestures and Lip Shape Integration for Cued Speech Recognition
ICPR '10: Proceedings of the 2010 20th International Conference on Pattern Recognition

In this article, automatic recognition of Cued Speech in French based on hidden Markov models (HMMs) is presented. Cued Speech is a visual mode, which uses hand shapes in different positions and in combination with lip-patterns of speech makes all the ...
Synergy of Lip-Motion and Acoustic Features in Biometric Speech and Speaker Recognition

This paper presents the scheme and evaluation of a robust audio-visual digit-and-speaker-recognition system using lip motion and speech biometrics. Moreover, a liveness verification barrier based on a person's lip movement is added to the system to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ICMI '00: Proceedings of the Third International Conference on Advances in Multimodal Interfaces

October 2000

673 pages

ISBN:3540411801

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 14 October 2000

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kelly DReilly Delannoy JMc Donald JMarkham CCrowley JIvanov YWren CGatica-Perez DJohnston MStiefelhagen R(2009)A framework for continuous multimodal sign language recognitionProceedings of the 2009 international conference on Multimodal interfaces10.1145/1647314.1647387(351-358)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1647314.1647387
Aran OBurger TCaplier AAkarun L(2009)A belief-based sequential fusion approach for fusing manual signs and non-manual signalsPattern Recognition10.1016/j.patcog.2008.09.01042:5(812-822)Online publication date: 1-May-2009
https://dl.acm.org/doi/10.1016/j.patcog.2008.09.010
Caplier AStillittano SAran OAkarun LBailly GBeautemps DAboutabit NBurger T(2007)Image and video for hearing impaired peopleJournal on Image and Video Processing10.1155/2007/456412007:5(1-14)Online publication date: 1-Jan-2007
https://dl.acm.org/doi/10.1155/2007/45641
Ong SRanganath S(2005)Automatic Sign Language AnalysisIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2005.11227:6(873-891)Online publication date: 1-Jun-2005
https://dl.acm.org/doi/10.1109/TPAMI.2005.112

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Multimodal speaker/speech recognition using lip motion, lip texture and audio

Gestures and Lip Shape Integration for Cued Speech Recognition

Synergy of Lip-Motion and Acoustic Features in Biometric Speech and Speaker Recognition

Comments

Information

Published In

Publisher

Publication History

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations