More Web Proxy on the site http://driver.im/

research-article

Structural and temporal inference search (STIS): pattern identification in multimodal data

Authors:

Chreston Miller,

Louis-Philippe Morency,

Francis QuekAuthors Info & Claims

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Pages 101 - 108

https://doi.org/10.1145/2388676.2388702

Published: 22 October 2012 Publication History

Abstract

There are a multitude of annotated behavior corpora (manual and automatic annotations) available as research expands in multimodal analysis of human behavior. Despite the rich representations within these datasets, search strategies are limited with respect to the advanced representations and complex structures describing human interaction sequences. The relationships amongst human interactions are structural in nature. Hence, we present Structural and Temporal Inference Search (STIS) to support search for relevant patterns within a multimodal corpus based on the structural and temporal nature of human interactions. The user defines the structure of a behavior of interest driving a search focused on the characteristics of the structure. Occurrences of the structure are returned. We compare against two pattern mining algorithms purposed for pattern identification amongst sequences of symbolic data (e.g., sequence of events such as behavior interactions). The results are promising as STIS performs well with several datasets.

References

[1]

J. F. Allen. Maintaining knowledge about temporal intervals. Commun. ACM, 26(11):832--843, 1983.

Digital Library

[2]

L. Breiman. Bagging predictors. Machine Learning, 24:123--140, 1996.

[3]

J. Carletta, et al. The ami meeting corpus: A pre-announcement. In MLMI, volume 3869 of LNCS, pages 28--39. Springer Berlin / Heidelberg, 2006.

[4]

L. Chen, et al. A multimodal analysis of floor control in meetings. In MLMI'06, pages 36--49.

Digital Library

[5]

L. Chen, et al. Vace multimodal meeting corpus. MLMI '06, pages 40--51.

Digital Library

[6]

EXMARaLDA. http://www.exmaralda.org, Last Checked: May, 2012.

[7]

C. Freksa. Temporal reasoning based on semi-intervals. Artificial Intelligence, 54(1--2):199--227, 1992.

Digital Library

[8]

J. Gratch, et al. Creating rapport with virtual agents. In Intelligent Virtual Agents, volume 4722 of LNCS, pages 125--138. Springer Berlin / Heidelberg, 2007.

Digital Library

[9]

J. Hagedorn, J. Hailpern, and K. G. Karahalios. Vcode and vdata: illustrating a new framework for supporting the video annotation workflow. In AVI '08, pages 317--321. ACM.

Digital Library

[10]

A. Kendon. Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press, 1990.

[11]

M. Kipp. Anvil - a generic annotation tool for multimodal dialogue. In Eurospeech, 2001.

[12]

M. Kipp. Spatiotemporal coding in anvil. In LREC, 2008.

[13]

S. Laxman and P. Sastry. A survey of temporal data mining. Sadhana, 31(2):173--198, 04 2006.

[14]

S. Laxman, P. Sastry, and K. Unnikrishnan. Discovering frequent episodes and learning hidden markov models: a formal connection. TKDE, 17(11):1505--1517, Nov. 2005.

Digital Library

[15]

M. Magnusson. Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior Research Methods, 32:93--110, 2000.

[16]

G. McKeown, et al. The semaine corpus of emotionally coloured character interactions. In ICME '10, pages 1079--1084.

[17]

D. McNeill. Gesture, gaze, and ground. In MLMI'06, volume 3869 of LNCS, pages 1--14.

[18]

D. McNeill. Hand and Mind: What gestures reveal about thought. Chicago: U. of Chicago Press, 1992.

[19]

D. McNeill et al. Mind-merging. In Expressing oneself / expressing one's self: Communication, language, cognition, and identity, 2007.

[20]

C. Miller and F. Quek. Toward multimodal situated analysis. In ICMI '11.

Digital Library

[21]

C. Miller and F. Quek. Interactive data-driven discovery of temporal behavior models from events in media streams. In ACM MM, 2012.

Digital Library

[22]

F. Mörchen. Unsupervised pattern mining from symbolic temporal data. SIGKDD Explor. Newsl., 9(1):41--55, 2007.

Digital Library

[23]

F. Mörchen and D. Fradkin. Robust mining of time intervals with semi-interval partial order patterns. In SIAM Conference on Data Mining (SDM), 2010.

[24]

L.-P. Morency, I. de Kok, and J. Gratch. Context-based recognition during human interactions: automatic feature selection and encoding dictionary. In ICMI '08, pages 181--188. ACM.

Digital Library

[25]

D. Patnaik, et al. Experiences with mining temporal event sequences from electronic medical records: initial successes and some challenges. In KDD '11, pages 360--368. ACM.

Digital Library

[26]

D. Patnaik, P. S. Sastry, and K. P. Unnikrishnan. Inferring neuronal network connectivity from spike data: A temporal data mining approach. Scientific Programming, 16(1):49--77, January 2007.

Digital Library

[27]

Qt. http://qt.nokia.com/, Last Checked: Mar., 2012.

[28]

F. Quek. The catchment feature model: a device for multimodal fusion and a bridge between signal and sense. EURASIP J. Appl. Signal Process., 2004:1619--1636, Jan. 2004.

Digital Library

[29]

F. Quek, T. Rose, and D. McNeill. Multimodal meeting analysis. In IA, 2005.

[30]

R. T. Rose, F. Quek, and Y. Shi. Macvissta: a system for multimodal analysis. In ICMI '04, pages 259--264.

Digital Library

[31]

P. S. Sastry and K. P. Unnikrishnan. Conditional probability based significance tests for sequential patterns in multi-neuronal spike trains. 2008.

[32]

T. Schmidt. The transcription system exmaralda: An application of the annotation graph formalism as the basis of a database of multilingual spoken discourse. In Proceedings of the IRCS Workshop On Linguistic Databases, pages 219--227, 2001.

[33]

T. Schmidt, et al. An exchange format for multimodal annotations. In Multimodal corpora, pages 207--221. Springer-Verlag, Berlin, Heidelberg, 2009.

[34]

T. Schmidt and K. Wörner. Exmaralda - creating, analysing and sharing spoken language corpora for pragmatic research. Pragmatics, 19, 2009.

[35]

Theme. http://www.noldus.com/human-behavior-research/products/theme.

[36]

M. Voit and R. Stiefelhagen. 3d user-perspective, voxel-based estimation of visual focus of attention in dynamic meeting scenarios. In ICMI-MLMI '10, pages 51:1--51:8. ACM, 2010.

Digital Library

[37]

U. Westermann and R. Jain. Toward a common event model for multimedia applications. IEEE MultiMedia, 14:19--29, January 2007.

Digital Library

[38]

P. Wittenburg, et al. Elan: a professional framework for multimodality research. In LREC, 2006.

Cited By

Okada SNguyen LAran OGatica-Perez D(2019)Modeling Dyadic and Group Impressions with Intermodal and Interperson FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/326575415:1s(1-30)Online publication date: 24-Jan-2019
https://dl.acm.org/doi/10.1145/3265754
Miller CMiller C(2019)Timing is Everything: Identifying Diverse Interaction Dynamics in Scenario and Non-Scenario Meetings2019 15th International Conference on eScience (eScience)10.1109/eScience.2019.00029(203-212)Online publication date: Sep-2019
https://doi.org/10.1109/eScience.2019.00029
Okada SAran OGatica-Perez DZhang ZCohen PBohus DHoraud RMeng H(2015)Personality Trait Classification via Co-Occurrent Multiparty Multimodal Event DiscoveryProceedings of the 2015 ACM on International Conference on Multimodal Interaction10.1145/2818346.2820757(15-22)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1145/2818346.2820757
Show More Cited By

Index Terms

Structural and temporal inference search (STIS): pattern identification in multimodal data
1. Computing methodologies
  1. Artificial intelligence
    1. Knowledge representation and reasoning
2. Theory of computation
  1. Logic
    1. Modal and temporal logics

Recommendations

Interactive relevance search and modeling: support for expert-driven analysis of multimodal data
ICMI '13: Proceedings of the 15th ACM on International conference on multimodal interaction

In this paper we present the findings of three longitudinal case studies in which a new method for conducting multimodal analysis of human behavior is tested. The focus of this new method is to engage a researcher integrally in the analysis process and ...
Interactive data-driven search and discovery of temporal behavior patterns from media streams
MM '12: Proceedings of the 20th ACM international conference on Multimedia

The presented thesis work addresses how social scientists may derive patterns of human behavior captured in media streams. Currently, media streams are being segmented into sequences of events describing the actions captured in the streams, such as the ...
Interactive data-driven discovery of temporal behavior models from events in media streams
MM '12: Proceedings of the 20th ACM international conference on Multimedia

This paper investigates a technique for the discovery of temporal behavior models within multimedia event data. Advancements in both technology and the marketplace present us the opportunity for research in analysis of situated human behavior using ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

October 2012

636 pages

ISBN:9781450314671

DOI:10.1145/2388676

General Chairs:
Louis-Philippe Morency
University of Southern California, USA
,
Dan Bohus
Microsoft Research, USA
,
Hamid Aghajan
Stanford University, USA
,
Program Chairs:
Justine Cassell
Carnegie Mellon University, USA
,
Anton Nijholt
University of Twente, Netherlands
,
Julien Epps
The University of New South Wales, Australia

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '12

Sponsor:

SIGCHI

ICMI '12: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 22 - 26, 2012

California, Santa Monica, USA

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
135
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Okada SNguyen LAran OGatica-Perez D(2019)Modeling Dyadic and Group Impressions with Intermodal and Interperson FeaturesACM Transactions on Multimedia Computing, Communications, and Applications10.1145/326575415:1s(1-30)Online publication date: 24-Jan-2019
https://dl.acm.org/doi/10.1145/3265754
Miller CMiller C(2019)Timing is Everything: Identifying Diverse Interaction Dynamics in Scenario and Non-Scenario Meetings2019 15th International Conference on eScience (eScience)10.1109/eScience.2019.00029(203-212)Online publication date: Sep-2019
https://doi.org/10.1109/eScience.2019.00029
Okada SAran OGatica-Perez DZhang ZCohen PBohus DHoraud RMeng H(2015)Personality Trait Classification via Co-Occurrent Multiparty Multimodal Event DiscoveryProceedings of the 2015 ACM on International Conference on Multimodal Interaction10.1145/2818346.2820757(15-22)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1145/2818346.2820757
Miller CQuek FMorency LKankanhalli MRueger SManmatha RJose Jvan Rijsbergen K(2014)Search Strategies for Pattern Identification in Multimodal DataProceedings of International Conference on Multimedia Retrieval10.1145/2578726.2578761(273-280)Online publication date: 1-Apr-2014
https://dl.acm.org/doi/10.1145/2578726.2578761
Miller CQuek FMorency LEpps JChen FOviatt SMase KSears AJokinen KSchuller B(2013)Interactive relevance search and modelingProceedings of the 15th ACM on International conference on multimodal interaction10.1145/2522848.2522889(149-156)Online publication date: 9-Dec-2013
https://dl.acm.org/doi/10.1145/2522848.2522889
Miller CBabaguchi NAizawa KSmith JSatoh SPlagemann THua XYan R(2012)Interactive data-driven search and discovery of temporal behavior patterns from media streamsProceedings of the 20th ACM international conference on Multimedia10.1145/2393347.2396512(1433-1436)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2393347.2396512

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents