More Web Proxy on the site http://driver.im/

research-article

Dynamic Word Embeddings for Evolving Semantic Discovery

Authors:

Hui XiongAuthors Info & Claims

WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

Pages 673 - 681

https://doi.org/10.1145/3159652.3159703

Published: 02 February 2018 Publication History

Abstract

Word evolution refers to the changing meanings and associations of words throughout time, as a byproduct of human language evolution. By studying word evolution, we can infer social trends and language constructs over different periods of human history. However, traditional techniques such as word representation learning do not adequately capture the evolving language structure and vocabulary. In this paper, we develop a dynamic statistical model to learn time-aware word vector representation. We propose a model that simultaneously learns time-aware embeddings and solves the resulting alignment problem. This model is trained on a crawled NYTimes dataset. Additionally, we develop multiple intuitive evaluation strategies of temporal word embeddings. Our qualitative and quantitative tests indicate that our method not only reliably captures this evolution over time, but also consistently outperforms state-of-the-art temporal embedding approaches on both semantic accuracy and alignment quality.

References

[1]

James Allan, Rahul Gupta, and Vikas Khandelwal . 2001. Temporal summaries of new topics. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 10--18.

Digital Library

[2]

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski . 2015. Rand-walk: A latent variable model approach to word embeddings. arXiv preprint arXiv:1502.03520 (2015).

[3]

Marco Baroni, Georgiana Dinu, and Germán Kruszewski . 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1). 238--247.

[4]

Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro . 2014. Analysing word meaning over time by exploiting temporal random indexing First Italian Conference on Computational Linguistics CLiC-it.

[5]

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin . 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research Vol. 3 (2003), 1137--1155.

Digital Library

[6]

David M Blei and John D Lafferty . 2006. Dynamic topic models Proceedings of the 23rd international conference on Machine learning. ACM, 113--120.

Digital Library

[7]

Hyunyoung Choi and Hal Varian . 2012. Predicting the present with Google Trends. Economic Record, Vol. 88, s1 (2012), 2--9.

[8]

Ronan Collobert and Jason Weston . 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning. ACM, 160--167.

Digital Library

[9]

Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman . 1990. Indexing by latent semantic analysis. Journal of the American society for information science, Vol. 41, 6 (1990), 391.

[10]

John R Firth . 1957. $$A synopsis of linguistic theory, 1930--1955$$. (1957).

[11]

Kristina Gulordava and Marco Baroni . 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics. Association for Computational Linguistics, 67--71.

Digital Library

[12]

William L Hamilton, Jure Leskovec, and Dan Jurafsky . 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. arXiv preprint arXiv:1605.09096 (2016).

[13]

Gerhard Heyer, Florian Holz, and Sven Teresniak . 2009. Change of Topics over Time-Tracking Topics by their Change of Meaning. KDIR Vol. 9 (2009), 223--228.

[14]

Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov . 2014. Temporal analysis of language through neural language models. arXiv preprint arXiv:1405.3515 (2014).

[15]

Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena . 2015. Statistically significant detection of linguistic change Proceedings of the 24th International Conference on World Wide Web. ACM, 625--635.

Digital Library

[16]

Matt J Kusner, Yu Sun, Nicholas I Kolkin, Kilian Q Weinberger, and others . 2015. From Word Embeddings To Document Distances. In ICML, Vol. Vol. 15. 957--966.

Digital Library

[17]

Omer Levy and Yoav Goldberg . 2014. Neural word embedding as implicit matrix factorization Advances in neural information processing systems. 2177--2185.

Digital Library

[18]

Omer Levy, Yoav Goldberg, and Ido Dagan . 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics Vol. 3 (2015), 211--225.

[19]

Xuanyi Liao and Guang Cheng . 2016. Analysing the Semantic Change Based on Word Embedding International Conference on Computer Processing of Oriental Languages. Springer, 213--223.

[20]

Kevin Lund and Curt Burgess . 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, Vol. 28, 2 (1996), 203--208.

[21]

Guy Merchant . 2001. Teenagers in cyberspace: An investigation of language use and language change in internet chatrooms. Journal of Research in Reading Vol. 24, 3 (2001), 293--306.

[22]

Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, and others . 2011. Quantitative analysis of culture using millions of digitized books. science, Vol. 331, 6014 (2011), 176--182.

[23]

Rada Mihalcea and Vivi Nastase . 2012. Word epoch disambiguation: Finding how words change over time Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, 259--263.

Digital Library

[24]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean . 2013 a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[25]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean . 2013 b. Distributed representations of words and phrases and their compositionality Advances in neural information processing systems. 3111--3119.

Digital Library

[26]

Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal . 2014. That's sick dude!: Automatic identification of word sense change across different timescales. arXiv preprint arXiv:1405.4392 (2014).

[27]

Jeffrey Pennington, Richard Socher, and Christopher D Manning . 2014. Glove: Global Vectors for Word Representation. In EMNLP, Vol. Vol. 14. 1532--1543.

[28]

James Petterson, Wray Buntine, Shravan M Narayanamurthy, Tibério S Caetano, and Alex J Smola . 2010. Word features for latent dirichlet allocation. In Advances in Neural Information Processing Systems. 1921--1929.

Digital Library

[29]

Michael JD Powell . 1973. On search directions for minimization algorithms. Mathematical Programming Vol. 4, 1 (1973), 193--201.

[30]

Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon . 2015. Collaborative filtering with graph information: Consistency and scalable methods Advances in neural information processing systems. 2107--2115.

Digital Library

[31]

Eyal Sagi, Stefan Kaufmann, and Brady Clark . 2011. Tracing semantic change with latent semantic analysis. Current methods in historical semantics (2011), 161--183.

[32]

Diane J Schiano, Coreena P Chen, Ellen Isaacs, Jeremy Ginsberg, Unnur Gretarsdottir, and Megan Huddleston . 2002. Teen use of messaging media. In CHI'02 extended abstracts on Human factors in computing systems. ACM, 594--595.

Digital Library

[33]

Ruben Sipos, Adith Swaminathan, Pannaga Shivaswamy, and Thorsten Joachims . 2012. Temporal corpus summarization using submodular word coverage Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 754--763.

Digital Library

[34]

Sali A Tagliamonte and Derek Denis . 2008. Linguistic ruin? LOL! Instant messaging and teen language. American speech, Vol. 83, 1 (2008), 3--34.

[35]

Xuri Tang, Weiguang Qu, and Xiaohe Chen . 2016. Semantic change computation: A successive approach. World Wide Web, Vol. 19, 3 (2016), 375--415.

Digital Library

[36]

Xuerui Wang and Andrew McCallum . 2006. Topics over time: a non-Markov continuous-time model of topical trends Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 424--433.

Digital Library

[37]

Derry Tanti Wijaya and Reyyan Yeniterzi . 2011. Understanding semantic change of words over centuries Proceedings of the 2011 international workshop on DETecting and Exploiting Cultural diversiTy on the social web. ACM, 35--40.

Digital Library

[38]

Stephen J Wright . 2015. Coordinate descent algorithms. Mathematical Programming Vol. 151, 1 (2015), 3--34.

Digital Library

[39]

Hsiang-Fu Yu, Cho-Jui Hsieh, Si Si, and Inderjit Dhillon . 2012. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems 12th IEEE International Conference on Data Mining (ICDM). IEEE, 765--774.

Digital Library

[40]

Yating Zhang, Adam Jatowt, Sourav S Bhowmick, and Katsumi Tanaka . 2016. The Past is Not a Foreign Country: Detecting Semantically Similar Terms across Time. IEEE Transactions on Knowledge and Data Engineering, Vol. 28, 10 (2016), 2793--2807.

Digital Library

Cited By

Couto MPerez AParapar JLosada D(2025)Temporal Word Embeddings for Early Detection of Psychological Disorders on Social MediaJournal of Healthcare Informatics Research10.1007/s41666-025-00186-9Online publication date: 22-Jan-2025
https://doi.org/10.1007/s41666-025-00186-9
Ling ZAida TOka TKomachi M(2024)Construction and Analysis of Evaluation Dataset for Japanese Lexical Semantic Change Detection日本語意味変化検出のための評価データセットの構築と分析Journal of Natural Language Processing10.5715/jnlp.31.148731:4(1487-1522)Online publication date: 2024
https://doi.org/10.5715/jnlp.31.1487
Asztalos BPalla GCzégel D(2024)Anomalous diffusion analysis of semantic evolution in major Indo-European languagesPLOS ONE10.1371/journal.pone.029865019:3(e0298650)Online publication date: 26-Mar-2024
https://doi.org/10.1371/journal.pone.0298650
Show More Cited By

Index Terms

Dynamic Word Embeddings for Evolving Semantic Discovery
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Jointly learning bilingual word embeddings and alignments
Abstract
Learning bilingual word embeddings can be much easier if the parallel corpora are available with their words well aligned explicitly. However, in most cases, the parallel corpora only provide a set of pairs that are semantically equivalent to each ...
An accurate transformer-based model for transition-based dependency parsing of free word order languages
Abstract
Transformer models are the state-of-the-art in Natural Language Processing (NLP) and the core of the Large Language Models (LLMs). We propose a transformer-based model for transition-based dependency parsing of free word order languages. We have ...
Composing Word Embeddings for Compound Words Using Linguistic Knowledge
In recent years, the use of distributed representations has been a fundamental technology for natural language processing. However, Japanese has multiple compound words, and often we must compare the meanings of a word and a compound word. Moreover, word ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

February 2018

821 pages

ISBN:9781450355810

DOI:10.1145/3159652

General Chairs:
Yi Chang
Jilin University, Huawei Inc.
,
Chengxiang Zhai
University of Illinois Urbana-Champaign
,
Program Chairs:
Yan Liu
University of Southern California
,
Yoelle Maarek
Amazon

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 February 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WSDM 2018

Sponsor:

WSDM 2018: The Eleventh ACM International Conference on Web Search and Data Mining

February 5 - 9, 2018

CA, Marina Del Rey, USA

Acceptance Rates

WSDM '18 Paper Acceptance Rate 81 of 514 submissions, 16%;

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

88
Total Citations
View Citations
2,023
Total Downloads

Downloads (Last 12 months)184
Downloads (Last 6 weeks)28

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Couto MPerez AParapar JLosada D(2025)Temporal Word Embeddings for Early Detection of Psychological Disorders on Social MediaJournal of Healthcare Informatics Research10.1007/s41666-025-00186-9Online publication date: 22-Jan-2025
https://doi.org/10.1007/s41666-025-00186-9
Ling ZAida TOka TKomachi M(2024)Construction and Analysis of Evaluation Dataset for Japanese Lexical Semantic Change Detection日本語意味変化検出のための評価データセットの構築と分析Journal of Natural Language Processing10.5715/jnlp.31.148731:4(1487-1522)Online publication date: 2024
https://doi.org/10.5715/jnlp.31.1487
Asztalos BPalla GCzégel D(2024)Anomalous diffusion analysis of semantic evolution in major Indo-European languagesPLOS ONE10.1371/journal.pone.029865019:3(e0298650)Online publication date: 26-Mar-2024
https://doi.org/10.1371/journal.pone.0298650
Dogan HNguyen KLourentzou I(2024)Narrative Characteristics in Refugee Discourse: An Analysis of American Public Opinion on the Afghan Refugee Crisis After the Taliban TakeoverProceedings of the ACM on Human-Computer Interaction10.1145/36537038:CSCW1(1-31)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3653703
Lin DCaba Heilbron FLee JWang OMartelaro N(2024)VideoMap: Supporting Video Exploration, Brainstorming, and Prototyping in the Latent SpaceProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656192(311-327)Online publication date: 23-Jun-2024
https://dl.acm.org/doi/10.1145/3635636.3656192
Buchmüller RKörte FKeim D(2024)Seeing the Shift: Keep an Eye on Semantic Changes in Times of LLMs2024 IEEE Visualization in Data Science (VDS)10.1109/VDS63897.2024.00010(40-47)Online publication date: 14-Oct-2024
https://doi.org/10.1109/VDS63897.2024.00010
Palamarchuk DWilliams LMayer BDanielson TFaust RDeschaine LNorth C(2024)Visualizing Temporal Topic Embeddings with a CompassIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614331:1(272-282)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456143
Tang SMeng ZLiang S(2024)Dynamic Co-Embedding Model for Temporal Attributed NetworksIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.319356435:3(3488-3502)Online publication date: Mar-2024
https://doi.org/10.1109/TNNLS.2022.3193564
García JFigueroa XVaca COrozco DBaquerizo-Neira GJimenez-Pazmino PFantoni C(2024)Identifying Citizen Interests During the COVID-19 Pandemic Using Context Change in Twitter Conversations2024 Tenth International Conference on eDemocracy & eGovernment (ICEDEG)10.1109/ICEDEG61611.2024.10702049(1-9)Online publication date: 24-Jun-2024
https://doi.org/10.1109/ICEDEG61611.2024.10702049
Soleymanian MMashayekhi HRahimi M(2024)An incremental clustering algorithm based on semantic conceptsKnowledge and Information Systems10.1007/s10115-024-02063-066:6(3303-3335)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10115-024-02063-0
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten