More Web Proxy on the site http://driver.im/

research-article

Challenges in Deep Learning for Multimodal Applications

Author:

Sayan GhoshAuthors Info & Claims

ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

Pages 611 - 615

https://doi.org/10.1145/2818346.2823313

Published: 09 November 2015 Publication History

Abstract

This consortium paper outlines a research plan for investigating deep learning techniques as applied to multimodal multi-task learning and multimodal fusion. We discuss our prior research results in this area, and how these results motivate us to explore more in this direction. We also define concrete steps of enquiry we wish to undertake as a short-term goal, and further outline some other challenges of multimodal learning using deep neural networks, such as inter and intra-modality synchronization, robustness to noise in modality data acquisition, and data insufficiency.

References

[1]

Ngiam, Jiquan, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. "Multimodal deep learning." In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 689696.

[2]

Srivastava, Nitish, and Ruslan R. Salakhutdinov. "Multimodal learning with deep boltzmann machines." In Advances in neural information processing systems, pp. 2222--2230. 2012.

[3]

Srivastava, Nitish, and Ruslan Salakhutdinov. "Learning representations for multimodal data with deep belief nets." In International Conference on Machine Learning Workshop. 2012.

[4]

Martínez, Héctor P., and Georgios N. Yannakakis. "Deep Multimodal Fusion: Combining Discrete Events and Continuous Signals." In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 34--41. ACM, 2014.

Digital Library

[5]

Kahou, Samira Ebrahimi, Christopher Pal, Xavier Bouthillier, Pierre Froumenty, Çaglar Gülçehre, Roland Memisevic, Pascal Vincent et al. "Combining modality specific deep neural networks for emotion recognition in video." In Proceedings of the 15th ACM on International conference on multimodal interaction, pp. 543--550. ACM, 2013.

Digital Library

[6]

Karpathy, Andrej, Armand Joulin, and Fei Fei F. Li. "Deep fragment embeddings for bidirectional image sentence mapping." In Advances in neural information processing systems, pp. 1889--1897. 2014.

[7]

Ghosh, Sayan, Moitreya Chatterjee, and Louis-Philippe Morency. "A multimodal context-based approach for distress assessment." In Proceedings of the 16th International Conference on Multimodal Interaction, pp. 240--246. ACM, 2014.

Digital Library

[8]

Wolpert, David H. "Stacked generalization." Neural networks 5, no. 2 (1992): 241--259.

Digital Library

[9]

Ghosh, Sayan, Eugene Laksana, Stefan Scherer, LouisPhilippe Morency, "A Multi-label Convolutional Neural Network Approach to Cross-Domain Action Unit Detection", accepted for publication in Proceedings of Affective Computing and Intelligent Interaction Conference, 2015

[10]

Gong, Yunchao, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. "Deep convolutional ranking for multilabel image annotation." arXiv preprint arXiv:1312.4894 (2013).

[11]

Mavadati, S. Mohammad, Mohammad H. Mahoor, Kevin Bartlett, Philip Trinh, and Jeffrey F. Cohn. "Disfa: A spontaneous facial action intensity database." IEEE Transactions on Affective Computing, 4, no. 2 (2013): 151--160.

Digital Library

[12]

Valstar, M., J. Girard, T. Almaev, Gary McKeown, Marc Mehu, Lijun Yin, Maja Pantic, and J. Cohn. "FERA 2015-second facial expression recognition and analysis challenge." Proc. IEEE ICFG (2015).

[13]

Lucey, Patrick, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. "The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression." In Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, pp. 94--101.

[14]

Natarajan, Pradeep, Shuang Wu, Shiv Vitaladevuni, Xiaodan Zhuang, Stavros Tsakalidis, Unsang Park, Rohit Prasad, and Premkumar Natarajan. "Multimodal feature fusion for robust event detection in web videos." In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012, pp. 1298--1305. IEEE, 2012.

Digital Library

[15]

Jia, Yangqing, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. "Caffe: Convolutional architecture for fast feature embedding." In Proceedings of the ACM Multimedia, pp. 675--678. ACM, 2014.

Digital Library

[16]

Collobert, Ronan, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. EPFL-REPORT-82802. IDIAP, 2002.

[17]

Jia, Yangqing, Mathieu Salzmann, and Trevor Darrell. "Factorized latent spaces with structured sparsity." In Advances in Neural Information Processing Systems, pp. 982--990. 2010.

[18]

Moon, Tood K. "The expectation-maximization algorithm." IEEE Signal processing magazine, 13, no. 6 (1996): 47--60.

[19]

Song, Yale, Daniel McDuff, Deepak Vasisht, and Ashish Kapoor. "Exploiting Sparsity and Cooccurrence Structure for Action Unit Recognition.", International Conference on Automatic Face and Gesture Recognition. May 2015.

[20]

Cummins, Nicholas, Stefan Scherer, Jarek Krajewski, Sebastian Schnieder, Julien Epps, and Thomas F. Quatieri. "A review of depression and suicide risk assessment using speech analysis." Speech Communication 71 pp. 10--49, 2015.

Digital Library

[21]

Pennebaker, James W., Martha E. Francis, and Roger J. Booth. "Linguistic inquiry and word count: LIWC 2001", Mahway: Lawrence Erlbaum Associates 71, 2001.

Cited By

Zhang YYin YZimmermann RWang GVaradarajan JNg S(2020)An Enhanced GAN Model for Automatic Satellite-to-Map Image ConversionIEEE Access10.1109/ACCESS.2020.30250088(176704-176716)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3025008
Pandya MShah PJardosh S(2019)Medical image diagnosis for disease detection: A deep learning approachU-Healthcare Monitoring Systems10.1016/B978-0-12-815370-3.00003-7(37-60)Online publication date: 2019
https://doi.org/10.1016/B978-0-12-815370-3.00003-7
Silva PLuz EZanlorensi LMenotti DMoreira G(2018)Multimodal Feature Level Fusion based on Particle Swarm Optimization with Deep Transfer Learning2018 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC.2018.8477817(1-8)Online publication date: Jul-2018
https://doi.org/10.1109/CEC.2018.8477817

Index Terms

Challenges in Deep Learning for Multimodal Applications
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Neural networks
2. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

A Review on Methods and Applications in Multimodal Deep Learning
Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the ...
Multimodal and Crossmodal Representation Learning from Textual and Visual Features with Bidirectional Deep Neural Networks for Video Hyperlinking
iV&L-MM '16: Proceedings of the 2016 ACM workshop on Vision and Language Integration Meets Multimedia Fusion

Video hyperlinking represents a classical example of multimodal problems. Common approaches to such problems are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially ...
Bidirectional Joint Representation Learning with Symmetrical Deep Neural Networks for Multimodal and Crossmodal Applications
ICMR '16: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval

Common approaches to problems involving multiple modalities (classification, retrieval, hyperlinking, etc.) are early fusion of the initial modalities and crossmodal translation from one modality to the other. Recently, deep neural networks, especially ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

November 2015

678 pages

ISBN:9781450339124

DOI:10.1145/2818346

General Chairs:
Zhengyou Zhang
Microsoft Research, USA
,
Phil Cohen
VoiceBox Technologies, USA
,
Program Chairs:
Dan Bohus
Microsoft Research, USA
,
Radu Horaud
INRIA Grenoble Rhone-Alpes, France
,
Helen Meng
Chinese University of Hong Kong, China

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

U.S. Army Research Laboratory

Conference

ICMI '15

Sponsor:

SIGCHI

ICMI '15: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 9 - 13, 2015

Washington, Seattle, USA

Acceptance Rates

ICMI '15 Paper Acceptance Rate 52 of 127 submissions, 41%;

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
610
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)3

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YYin YZimmermann RWang GVaradarajan JNg S(2020)An Enhanced GAN Model for Automatic Satellite-to-Map Image ConversionIEEE Access10.1109/ACCESS.2020.30250088(176704-176716)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3025008
Pandya MShah PJardosh S(2019)Medical image diagnosis for disease detection: A deep learning approachU-Healthcare Monitoring Systems10.1016/B978-0-12-815370-3.00003-7(37-60)Online publication date: 2019
https://doi.org/10.1016/B978-0-12-815370-3.00003-7
Silva PLuz EZanlorensi LMenotti DMoreira G(2018)Multimodal Feature Level Fusion based on Particle Swarm Optimization with Deep Transfer Learning2018 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC.2018.8477817(1-8)Online publication date: Jul-2018
https://doi.org/10.1109/CEC.2018.8477817

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents