More Web Proxy on the site http://driver.im/

research-article

A Deep Multi-Modal Fusion Approach for Semantic Place Prediction in Social Media

Authors:

Zhongxuan LuoAuthors Info & Claims

MUSA2 '17: Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes

Pages 31 - 37

https://doi.org/10.1145/3132515.3132519

Published: 27 October 2017 Publication History

Abstract

Semantic places such as "home," "work," and "school" are much easier to understand compared to GPS coordinates or street addresses and contribute to the automatic inference of related activities, which could further help in the study of personal lifestyle patterns and the provision of more customized services for human beings. In this work, we present a feature-level fusion method for semantic place prediction that utilizes user-generated text-image pairs from online social media as input. To take full advantage of each specific modality, we concatenate features from two state-of-the-art Convolutional Neural Networks (CNNs) and train them together. To the best of our knowledge, the present study is the first attempt to conduct semantic place prediction based only on microblogging multimedia content. The experimental results demonstrate that our deep multi-modal architecture outperforms single-modal methods and the traditional fusion method.

References

[1]

Zhiyuan Cheng, James Caverlee, Krishna Yeswanth Kamath, and Kyumin Lee. 2011. Toward traffic-driven location-based web search. Proceedings of the 20th ACM international conference on Information and knowledge management. ACM, 805--814.

Digital Library

[2]

Zhiyuan Cheng, James Caverlee, and Kyumin Lee. 2010. You are where you tweet: a content-based approach to geo-locating twitter users Proceedings of the 19th ACM international conference on Information and knowledge management. ACM, 759--768.

Digital Library

[3]

Yue Gao, Yi Zhen, Haojie Li, and Tat-Seng Chua. 2016. Filtering of brand-related microblogs using social-smooth multiview embedding. IEEE Transactions on Multimedia Vol. 18, 10 (2016), 2115--2126.

[4]

Richang Hong, Yang Yang, Meng Wang, and Xian-Sheng Hua. 2015. Learning visual semantic relationships for efficient visual retrieval. IEEE Transactions on Big Data Vol. 1, 4 (2015), 152--161.

Digital Library

[5]

Chi-Min Huang, Josh Jia-Ching Ying, and Vincent S. Tseng. 2012. Mining users' behaviors and environments for semantic place prediction Nokia Mobile Data Challenge 2012 Workshop. p. Dedicated task, Vol. 1.

[6]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678.

Digital Library

[7]

Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014).

[8]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks Advances in neural information processing systems. 1097--1105.

Digital Library

[9]

John Krumm and Dany Rouhana. 2013. Placer: semantic place labels from diary data. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. ACM, 163--172.

Digital Library

[10]

John Krumm, Dany Rouhana, and Ming-Wei Chang. 2015. Placer++: Semantic place labels beyond the visit. Pervasive Computing and Communications (PerCom), 2015 IEEE International Conference on. IEEE, 11--19.

[11]

Juha K. Laurila, Daniel Gatica-Perez, Imad Aad, Olivier Bornet, Trinh-Minh-Tri Do, Olivier Dousse, Julien Eberle, Markus Miettinen, et al. 2012. The mobile data challenge: Big data for mobile computing research Pervasive Computing.

[12]

Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents Proceedings of the 31st International Conference on Machine Learning (ICML-14). 1188--1196.

Digital Library

[13]

Haojie Li, Yue Guan, Lijuan Liu, Fanglin Wang, and Ling Wang. 2016. Re-ranking for microblog retrieval via multiple graph model. Multimedia Tools and Applications Vol. 75, 15 (2016), 8939--8954.

Digital Library

[14]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[15]

Raul Montoliu, Adolfo Martínez-Uso, Jose Martínez-Sotoca, and J. McInerney. 2012. Semantic place prediction by combining smart binary classifiers Nokia Mobile Data Challenge 2012 Workshop. p. Dedicated task, Vol. Vol. 1.

[16]

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP, Vol. Vol. 14. 1532--1543.

[17]

Tatiana Pontes, Gabriel Magno, Marisa Vasconcelos, Aditi Gupta, Jussara Almeida, Ponnurangam Kumaraguru, and Virgilio Almeida. 2012. Beware of what you share: Inferring home location in social networks Data Mining Workshops (ICDMW), 2012 IEEE 12th International Conference on. IEEE, 571--578.

Digital Library

[18]

Jinhui Tang, Xiangbo Shu, Zechao Li, Guo-Jun Qi, and Jingdong Wang. 2016. Generalized deep transfer networks for knowledge propagation in heterogeneous domains. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 12, 4s (2016), 68.

Digital Library

[19]

Ying Zhu, Yong Sun, Yu Wang. 2012. Nokia mobile data challenge: Predicting semantic place and next place via mobile data. Work, Vol. 80, 100 (2012), 120.

[20]

Mao Ye, Dong Shou, Wang-Chien Lee, Peifeng Yin, and Krzysztof Janowicz. 2011. On the semantic annotation of places in location-based social networks Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 520--528.

Digital Library

[21]

Ye Zhang and Byron Wallace. 2015. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820 (2015).

[22]

Danning Zheng, Tianran Hu, Quanzeng You, Henry A. Kautz, and Jiebo Luo. 2015. Towards Lifestyle Understanding: Predicting Home and Vacation Locations from User's Online Photo Collections. In ICWSM. 553--561.

[23]

Yin Zhu, Erheng Zhong, Zhongqi Lu, and Qiang Yang. 2012. Feature engineering for place category classification Workshop on the Nokia Mobile Data Challenge.

Cited By

Carvalho Boaro JBezerra da Costa Pde Sousa Moraes DCutrim dos Santos PRibeiro JDuarte JSardinha AColcher S(2024)A Middleware Architecture for Enhancing Multimedia Flows with High-Level Semantic InformationProceedings of the 2024 ACM International Conference on Interactive Media Experiences Workshops10.1145/3672406.3672407(1-6)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3672406.3672407
Zhang DXu RHuang WZhao KChen MElkind E(2023)Towards an integrated view of semantic annotation for POIs with spatial and textual informationProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/271(2441-2449)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/271
Xu RChen MGong YLiu YYu XNie L(2023)TME: Tree-guided Multi-task Embedding Learning towards Semantic Venue AnnotationACM Transactions on Information Systems10.1145/358255341:4(1-24)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1145/3582553
Show More Cited By

Recommendations

Robust Deep Multi-modal Learning Based on Gated Information Fusion Network
Computer Vision – ACCV 2018
Abstract
The goal of multi-modal learning is to use complementary information on the relevant task provided by the multiple modalities to achieve reliable and robust performance. Recently, deep learning has led significant improvement in multi-modal ...
Uses and gratifications of social networking sites for bridging and bonding social capital

Applying uses and gratifications theory (UGT) and social capital theory, our study examined users of four social networking sites (SNSs) (Facebook, Twitter, Instagram, and Snapchat), and their influence on online bridging and bonding social capital. ...
Multi-focus image fusion with a deep convolutional neural network

Introduces Convolutional neural networks (CNNs) into the field of image fusion.Discusses the feasibility and superiority of CNNs used for image fusion.Proposes a state-of-the-art CNN-based multi-focus image fusion method.Exhibits the potential of CNNs ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MUSA2 '17: Proceedings of the Workshop on Multimodal Understanding of Social, Affective and Subjective Attributes

October 2017

78 pages

ISBN:9781450355094

DOI:10.1145/3132515

General Chairs:
Xavier Alameda-Pineda
INRIA, France
,
Miriam Redi
Bell Labs, UK
,
Mohammed Soleymani
University of Geneva, Switzerland
,
Nicu Sebe
University of Trento, Italy
,
Shih-Fu Chang
Columbia University, USA
,
Samuel D. Gosling
University of Texas, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Funds of China

Conference

MM '17

Sponsor:

SIGMM

MM '17: ACM Multimedia Conference

October 27, 2017

California, Mountain View, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
123
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Carvalho Boaro JBezerra da Costa Pde Sousa Moraes DCutrim dos Santos PRibeiro JDuarte JSardinha AColcher S(2024)A Middleware Architecture for Enhancing Multimedia Flows with High-Level Semantic InformationProceedings of the 2024 ACM International Conference on Interactive Media Experiences Workshops10.1145/3672406.3672407(1-6)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3672406.3672407
Zhang DXu RHuang WZhao KChen MElkind E(2023)Towards an integrated view of semantic annotation for POIs with spatial and textual informationProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/271(2441-2449)Online publication date: 19-Aug-2023
https://dl.acm.org/doi/10.24963/ijcai.2023/271
Xu RChen MGong YLiu YYu XNie L(2023)TME: Tree-guided Multi-task Embedding Learning towards Semantic Venue AnnotationACM Transactions on Information Systems10.1145/358255341:4(1-24)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1145/3582553
Rimjhim Dandapat S(2023)Tagging Multi-Label Categories to Points of Interest From Check-In DataIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2022.32293387:4(1191-1204)Online publication date: Aug-2023
https://doi.org/10.1109/TETCI.2022.3229338
Xu HXu RChen MLiu YYu X(2022)CAVE-SC: Inferring categories for venues using check-insInformation Sciences10.1016/j.ins.2022.08.056611(159-172)Online publication date: Sep-2022
https://doi.org/10.1016/j.ins.2022.08.056
Wang NLiu BNiu MMeng KLi HLiu BWang Z(2021)Semantic Place Prediction With User Attribute in Social MediaIEEE MultiMedia10.1109/MMUL.2021.308971928:4(29-37)Online publication date: 1-Oct-2021
https://doi.org/10.1109/MMUL.2021.3089719

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents