[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3159652.3159703acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article

Dynamic Word Embeddings for Evolving Semantic Discovery

Published: 02 February 2018 Publication History

Abstract

Word evolution refers to the changing meanings and associations of words throughout time, as a byproduct of human language evolution. By studying word evolution, we can infer social trends and language constructs over different periods of human history. However, traditional techniques such as word representation learning do not adequately capture the evolving language structure and vocabulary. In this paper, we develop a dynamic statistical model to learn time-aware word vector representation. We propose a model that simultaneously learns time-aware embeddings and solves the resulting alignment problem. This model is trained on a crawled NYTimes dataset. Additionally, we develop multiple intuitive evaluation strategies of temporal word embeddings. Our qualitative and quantitative tests indicate that our method not only reliably captures this evolution over time, but also consistently outperforms state-of-the-art temporal embedding approaches on both semantic accuracy and alignment quality.

References

[1]
James Allan, Rahul Gupta, and Vikas Khandelwal . 2001. Temporal summaries of new topics. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 10--18.
[2]
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski . 2015. Rand-walk: A latent variable model approach to word embeddings. arXiv preprint arXiv:1502.03520 (2015).
[3]
Marco Baroni, Georgiana Dinu, and Germán Kruszewski . 2014. Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors. In ACL (1). 238--247.
[4]
Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro . 2014. Analysing word meaning over time by exploiting temporal random indexing First Italian Conference on Computational Linguistics CLiC-it.
[5]
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin . 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research Vol. 3 (2003), 1137--1155.
[6]
David M Blei and John D Lafferty . 2006. Dynamic topic models Proceedings of the 23rd international conference on Machine learning. ACM, 113--120.
[7]
Hyunyoung Choi and Hal Varian . 2012. Predicting the present with Google Trends. Economic Record, Vol. 88, s1 (2012), 2--9.
[8]
Ronan Collobert and Jason Weston . 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning. ACM, 160--167.
[9]
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman . 1990. Indexing by latent semantic analysis. Journal of the American society for information science, Vol. 41, 6 (1990), 391.
[10]
John R Firth . 1957. $$A synopsis of linguistic theory, 1930--1955$$. (1957).
[11]
Kristina Gulordava and Marco Baroni . 2011. A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics. Association for Computational Linguistics, 67--71.
[12]
William L Hamilton, Jure Leskovec, and Dan Jurafsky . 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. arXiv preprint arXiv:1605.09096 (2016).
[13]
Gerhard Heyer, Florian Holz, and Sven Teresniak . 2009. Change of Topics over Time-Tracking Topics by their Change of Meaning. KDIR Vol. 9 (2009), 223--228.
[14]
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov . 2014. Temporal analysis of language through neural language models. arXiv preprint arXiv:1405.3515 (2014).
[15]
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena . 2015. Statistically significant detection of linguistic change Proceedings of the 24th International Conference on World Wide Web. ACM, 625--635.
[16]
Matt J Kusner, Yu Sun, Nicholas I Kolkin, Kilian Q Weinberger, and others . 2015. From Word Embeddings To Document Distances. In ICML, Vol. Vol. 15. 957--966.
[17]
Omer Levy and Yoav Goldberg . 2014. Neural word embedding as implicit matrix factorization Advances in neural information processing systems. 2177--2185.
[18]
Omer Levy, Yoav Goldberg, and Ido Dagan . 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics Vol. 3 (2015), 211--225.
[19]
Xuanyi Liao and Guang Cheng . 2016. Analysing the Semantic Change Based on Word Embedding International Conference on Computer Processing of Oriental Languages. Springer, 213--223.
[20]
Kevin Lund and Curt Burgess . 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, Vol. 28, 2 (1996), 203--208.
[21]
Guy Merchant . 2001. Teenagers in cyberspace: An investigation of language use and language change in internet chatrooms. Journal of Research in Reading Vol. 24, 3 (2001), 293--306.
[22]
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, and others . 2011. Quantitative analysis of culture using millions of digitized books. science, Vol. 331, 6014 (2011), 176--182.
[23]
Rada Mihalcea and Vivi Nastase . 2012. Word epoch disambiguation: Finding how words change over time Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, 259--263.
[24]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean . 2013 a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
[25]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean . 2013 b. Distributed representations of words and phrases and their compositionality Advances in neural information processing systems. 3111--3119.
[26]
Sunny Mitra, Ritwik Mitra, Martin Riedl, Chris Biemann, Animesh Mukherjee, and Pawan Goyal . 2014. That's sick dude!: Automatic identification of word sense change across different timescales. arXiv preprint arXiv:1405.4392 (2014).
[27]
Jeffrey Pennington, Richard Socher, and Christopher D Manning . 2014. Glove: Global Vectors for Word Representation. In EMNLP, Vol. Vol. 14. 1532--1543.
[28]
James Petterson, Wray Buntine, Shravan M Narayanamurthy, Tibério S Caetano, and Alex J Smola . 2010. Word features for latent dirichlet allocation. In Advances in Neural Information Processing Systems. 1921--1929.
[29]
Michael JD Powell . 1973. On search directions for minimization algorithms. Mathematical Programming Vol. 4, 1 (1973), 193--201.
[30]
Nikhil Rao, Hsiang-Fu Yu, Pradeep K Ravikumar, and Inderjit S Dhillon . 2015. Collaborative filtering with graph information: Consistency and scalable methods Advances in neural information processing systems. 2107--2115.
[31]
Eyal Sagi, Stefan Kaufmann, and Brady Clark . 2011. Tracing semantic change with latent semantic analysis. Current methods in historical semantics (2011), 161--183.
[32]
Diane J Schiano, Coreena P Chen, Ellen Isaacs, Jeremy Ginsberg, Unnur Gretarsdottir, and Megan Huddleston . 2002. Teen use of messaging media. In CHI'02 extended abstracts on Human factors in computing systems. ACM, 594--595.
[33]
Ruben Sipos, Adith Swaminathan, Pannaga Shivaswamy, and Thorsten Joachims . 2012. Temporal corpus summarization using submodular word coverage Proceedings of the 21st ACM international conference on Information and knowledge management. ACM, 754--763.
[34]
Sali A Tagliamonte and Derek Denis . 2008. Linguistic ruin? LOL! Instant messaging and teen language. American speech, Vol. 83, 1 (2008), 3--34.
[35]
Xuri Tang, Weiguang Qu, and Xiaohe Chen . 2016. Semantic change computation: A successive approach. World Wide Web, Vol. 19, 3 (2016), 375--415.
[36]
Xuerui Wang and Andrew McCallum . 2006. Topics over time: a non-Markov continuous-time model of topical trends Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 424--433.
[37]
Derry Tanti Wijaya and Reyyan Yeniterzi . 2011. Understanding semantic change of words over centuries Proceedings of the 2011 international workshop on DETecting and Exploiting Cultural diversiTy on the social web. ACM, 35--40.
[38]
Stephen J Wright . 2015. Coordinate descent algorithms. Mathematical Programming Vol. 151, 1 (2015), 3--34.
[39]
Hsiang-Fu Yu, Cho-Jui Hsieh, Si Si, and Inderjit Dhillon . 2012. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems 12th IEEE International Conference on Data Mining (ICDM). IEEE, 765--774.
[40]
Yating Zhang, Adam Jatowt, Sourav S Bhowmick, and Katsumi Tanaka . 2016. The Past is Not a Foreign Country: Detecting Semantically Similar Terms across Time. IEEE Transactions on Knowledge and Data Engineering, Vol. 28, 10 (2016), 2793--2807.

Cited By

View all
  • (2025)Temporal Word Embeddings for Early Detection of Psychological Disorders on Social MediaJournal of Healthcare Informatics Research10.1007/s41666-025-00186-9Online publication date: 22-Jan-2025
  • (2024)Construction and Analysis of Evaluation Dataset for Japanese Lexical Semantic Change Detection日本語意味変化検出のための評価データセットの構築と分析Journal of Natural Language Processing10.5715/jnlp.31.148731:4(1487-1522)Online publication date: 2024
  • (2024)Anomalous diffusion analysis of semantic evolution in major Indo-European languagesPLOS ONE10.1371/journal.pone.029865019:3(e0298650)Online publication date: 26-Mar-2024
  • Show More Cited By

Index Terms

  1. Dynamic Word Embeddings for Evolving Semantic Discovery

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining
    February 2018
    821 pages
    ISBN:9781450355810
    DOI:10.1145/3159652
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 February 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. dynamic word embeddings
    2. word semantic analysis

    Qualifiers

    • Research-article

    Conference

    WSDM 2018

    Acceptance Rates

    WSDM '18 Paper Acceptance Rate 81 of 514 submissions, 16%;
    Overall Acceptance Rate 498 of 2,863 submissions, 17%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)184
    • Downloads (Last 6 weeks)28
    Reflects downloads up to 25 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Temporal Word Embeddings for Early Detection of Psychological Disorders on Social MediaJournal of Healthcare Informatics Research10.1007/s41666-025-00186-9Online publication date: 22-Jan-2025
    • (2024)Construction and Analysis of Evaluation Dataset for Japanese Lexical Semantic Change Detection日本語意味変化検出のための評価データセットの構築と分析Journal of Natural Language Processing10.5715/jnlp.31.148731:4(1487-1522)Online publication date: 2024
    • (2024)Anomalous diffusion analysis of semantic evolution in major Indo-European languagesPLOS ONE10.1371/journal.pone.029865019:3(e0298650)Online publication date: 26-Mar-2024
    • (2024)Narrative Characteristics in Refugee Discourse: An Analysis of American Public Opinion on the Afghan Refugee Crisis After the Taliban TakeoverProceedings of the ACM on Human-Computer Interaction10.1145/36537038:CSCW1(1-31)Online publication date: 26-Apr-2024
    • (2024)VideoMap: Supporting Video Exploration, Brainstorming, and Prototyping in the Latent SpaceProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656192(311-327)Online publication date: 23-Jun-2024
    • (2024)Seeing the Shift: Keep an Eye on Semantic Changes in Times of LLMs2024 IEEE Visualization in Data Science (VDS)10.1109/VDS63897.2024.00010(40-47)Online publication date: 14-Oct-2024
    • (2024)Visualizing Temporal Topic Embeddings with a CompassIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345614331:1(272-282)Online publication date: 10-Sep-2024
    • (2024)Dynamic Co-Embedding Model for Temporal Attributed NetworksIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2022.319356435:3(3488-3502)Online publication date: Mar-2024
    • (2024)Identifying Citizen Interests During the COVID-19 Pandemic Using Context Change in Twitter Conversations2024 Tenth International Conference on eDemocracy & eGovernment (ICEDEG)10.1109/ICEDEG61611.2024.10702049(1-9)Online publication date: 24-Jun-2024
    • (2024)An incremental clustering algorithm based on semantic conceptsKnowledge and Information Systems10.1007/s10115-024-02063-066:6(3303-3335)Online publication date: 1-Jun-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media