short-paper

Investigating the use of visual focus of attention for audio-visual speaker diarisation

Authors:

Giulia Garau,

Sileye Ba,

Hervé Bourlard,

Jean-Marc OdobezAuthors Info & Claims

MM '09: Proceedings of the 17th ACM international conference on Multimedia

Pages 681 - 684

https://doi.org/10.1145/1631272.1631387

Published: 19 October 2009 Publication History

Get Access

Abstract

Audio-visual speaker diarisation is the task of estimating ``who spoke when'' using audio and visual cues.

In this paper we propose the combination of an audio diarisation system with psychology inspired visual features, reporting experiments on multiparty meetings, a challenging domain characterised by unconstrained interaction and participant movements.

More precisely the role of gaze in coordinating speaker turns was exploited by the use of Visual Focus of Attention features. Experiments were performed both with the reference and 3 automatic VFoA estimation systems, based on head pose and visual activity cues, of increasing complexity. VFoA features yielded consistent speaker diarisation improvements in combination with audio features using a multi-stream approach.

References

[1]

X. Anguera, C. Wooters, and J. Hernando. Speaker diarization for multi-party meetings using acoustic fusion. In Proc. ASRU, 2005.

Crossref

Google Scholar

[2]

S. Ba, H. Hung, and J.-M. Odobez. Visual Activity Context for Focus of Attention Estimation in Dynamic Meetings. In Proc. of ICME, 2009.

Digital Library

Google Scholar

[3]

J. Carletta et al. The AMI Meeting Corpus: A Pre-Announcement. Proc. MLMI, 2005.

Digital Library

Google Scholar

[4]

G. Friedland, H. Hung, and C. Yeo. Multi-Modal Speaker Diarization of Real-World Meetings using Compressed Domain Video Features. In Proc. ICASSP, 2009.

Digital Library

Google Scholar

[5]

D. Gatica--Perez. Automatic nonverbal analysis of social interaction in small groups: a review. Image and Vision Computing, Special Issue on Human Naturalistic Behavior, In press, 2009.

Digital Library

Google Scholar

[6]

A. Noulas and B. Krose. On-Line Multi-Modal Speaker Diarisation. In Proc. ICMI, 2007.

Digital Library

Google Scholar

[7]

D. Novick, B. Hansen, and K. Ward. Coordinating Turn-Taking with Gaze. In Proc. ICSLP, 1996.

Crossref

Google Scholar

[8]

K. Otsuka et al. A Realtime Multimodal System for Analysing Group Meetings by Combining Face Pose Tracking and Speaker Diarisation. Proc. ICMI, 2008.

Digital Library

Google Scholar

[9]

E. Padilha and J. Carletta. Nonverbal Behaviours Improving a Simulation of Small Group Discussion. In Proc. of the 1st Nordic Symposium on Multimodal Communications, pages 93--105, 2003.

Google Scholar

[10]

S. Tranter and D. Reynolds. An Overview of Automatic Speaker Diarization Systems. IEEE Transactions on Audio, Speech and Language Processing, 14, 2006.

Digital Library

Google Scholar

[11]

R. Vertegaal, R. Slagter, G. Van der Veer, and A. Nijholt. Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In Proc. of ACM SIGCHI, 2001.

Digital Library

Google Scholar

[12]

C. Wooters and M. Huijbregts. The ICSI RT07s Speaker Diarization System. Proc. Rich Transcription Spring Meeting Recognition Evaluation, 2007.

Google Scholar

[13]

M. Zobl, F. Wallhoff, and G. Rigoll. Action Recognition in Meeting Scenarios using Global Motion Features. In Proc. PETS-ICVS, 2003.

Google Scholar

Cited By

View all

Zhang YLiang SYang SLiu XWu ZShan SChen XShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)UniCon: Unified Context Network for Robust Active Speaker DetectionProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475275(3964-3972)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475275
Friedland GYeo CHung H(2010)DialocalizationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/1865106.18651116:4(1-18)Online publication date: 26-Nov-2010
https://dl.acm.org/doi/10.1145/1865106.1865111
Garau GBourlard H(2010)Using audio and visual cues for speaker diarisation initialisation2010 IEEE International Conference on Acoustics, Speech and Signal Processing10.1109/ICASSP.2010.5495101(4942-4945)Online publication date: Mar-2010
https://doi.org/10.1109/ICASSP.2010.5495101

Index Terms

Investigating the use of visual focus of attention for audio-visual speaker diarisation
1. Information systems
  1. Information retrieval
    1. Document representation
    2. Search engine architectures and scalability
      1. Search engine indexing

Recommendations

A Paralinguistic Approach To Speaker Diarisation: Using Age, Gender, Voice Likability and Personality Traits
MM '17: Proceedings of the 25th ACM international conference on Multimedia

In this work, we present a new view on automatic speaker diarisation, i.e., assessing "who speaks when", based on the recognition of speaker traits such as age, gender, voice likability, and personality. Traditionally, speaker diarisation is ...
Audio-Visual Speaker Identification: Hybrid Fusion Based Speaker Identification System for Human-Computer Interaction
Audio-visual prosody: perception, detection, and synthesis of prominence
Proceedings of the Third COST 2102 international training school conference on Toward autonomous, adaptive, and context-aware multimodal interfaces: theoretical and practical issues

In this chapter, we investigate the effects of facial prominence cues, in terms of gestures, when synthesized on animated talking heads. In the first study a speech intelligibility experiment is conducted, where speech quality is acoustically degraded, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

MM '09: Proceedings of the 17th ACM international conference on Multimedia

October 2009

1202 pages

ISBN:9781605586083

DOI:10.1145/1631272

General Chairs:
Wen Gao
Peking University, China
,
Yong Rui
Microsoft, China
,
Alan Hanjalic
Delft University of Technology, The Netherlands
,
Program Chairs:
Changsheng Xu
Institute of Automation, Chinese Academy of Sciences, China
,
Eckehard Steinbach
Technical University of Munich, Germany
,
Abdulmotaleb El Saddik
University of Ottawa, Canada
,
Michelle Zhou
IBM T. J. Watson Research Center, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

MM09

Sponsor:

SIGMM

MM09: ACM Multimedia Conference

October 19 - 24, 2009

Beijing, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
138
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhang YLiang SYang SLiu XWu ZShan SChen XShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)UniCon: Unified Context Network for Robust Active Speaker DetectionProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475275(3964-3972)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475275
Friedland GYeo CHung H(2010)DialocalizationACM Transactions on Multimedia Computing, Communications, and Applications10.1145/1865106.18651116:4(1-18)Online publication date: 26-Nov-2010
https://dl.acm.org/doi/10.1145/1865106.1865111
Garau GBourlard H(2010)Using audio and visual cues for speaker diarisation initialisation2010 IEEE International Conference on Acoustics, Speech and Signal Processing10.1109/ICASSP.2010.5495101(4942-4945)Online publication date: Mar-2010
https://doi.org/10.1109/ICASSP.2010.5495101

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A Paralinguistic Approach To Speaker Diarisation: Using Age, Gender, Voice Likability and Personality Traits

Audio-Visual Speaker Identification: Hybrid Fusion Based Speaker Identification System for Human-Computer Interaction

Audio-visual prosody: perception, detection, and synthesis of prominence