Abstract
Most knowledge graph completion (KGC) methods rely solely on structural information, even though a large number of publicly available KGs contain additional temporal (validity time intervals) and textual data (entity descriptions). While recent temporal KGC methods utilize time information to enhance link prediction, they do not leverage textual descriptions or support inductive inference (prediction for entities that have not been seen during training).
In this work, we propose a novel framework called TEMT that exploits the power of pre-trained language models (PLMs) for temporal KGC. TEMT predicts time intervals of facts by fusing their textual and temporal information. It also supports inductive inference by utilizing PLMs. In order to showcase the power of TEMT, we carry out several experiments including time interval prediction, both in transductive and inductive settings, and triple classification. The experimental results demonstrate that TEMT is competitive with the state-of-the-art, while also supporting inductiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The datasets and the source code are available at https://github.com/duyguislakoglu/TEMT.
- 2.
The name of the model used is all-mpnet-base-v2.
References
Agarap, A.F.: Deep learning using rectified linear units (ReLU). ArXiv abs/1803.08375 (2018)
Alam, M.M., et al.: Language model guided knowledge graph embeddings. IEEE Access 10, 76008–76020 (2022)
AlKhamissi, B., Li, M., Celikyilmaz, A., Diab, M.T., Ghazvininejad, M.: A review on language models as knowledge bases. ArXiv abs/2204.06031 (2022)
Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. Neural Inf. Process. Syst. (2013)
Cai, B., Xiang, Y., Gao, L., Zhang, H., Li, Y., Li, J.: Temporal knowledge graph completion: a survey. In: International Joint Conference on Artificial Intelligence (2022)
Cai, L., Janowicz, K., Yan, B., Zhu, R., Mai, G.: Time in a box: advancing knowledge graph completion with temporal scopes. In: Proceedings of the 11th Knowledge Capture Conference (2021)
Chen, Z., Xu, C., Su, F., Huang, Z., Dou, Y.: Incorporating structured sentences with time-enhanced Bert for fully-inductive temporal relation prediction. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (2023)
Dasgupta, S.S., Ray, S.N., Talukdar, P.P.: Hyte: hyperplane-based temporally aware knowledge graph embedding. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2018)
Daza, D., Cochez, M., Groth, P.T.: Inductive entity representations from text via link prediction. In: Proceedings of the Web Conference 2021 (2020)
Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2d knowledge graph embeddings. In: AAAI Conference on Artificial Intelligence (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (2019)
Gu, K., Budhkar, A.: A package for learning on tabular and text data with transformers. In: MAIWORKSHOP (2021)
Han, Z., et al.: Enhanced temporal knowledge embeddings with contextualized language representations. ArXiv abs/2203.09590 (2022)
Jain, P., Rathi, S., Mausam, Chakrabarti, S.: Temporal knowledge base completion: new algorithms and evaluation protocols. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)
Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: representation, acquisition, and applications. IEEE Trans. Neural Netw. Learn. Syst. (2020)
Jia, Z., Pramanik, S., Roy, R.S., Weikum, G.: Complex temporal question answering on knowledge graphs. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
Lacroix, T., Obozinski, G., Usunier, N.: Tensor decompositions for temporal knowledge base completion. In: International Conference on Learning Representations (ICLR) (2020)
Leblay, J., Chekol, M.W.: Deriving validity time in knowledge graph. In: Companion Proceedings of the The Web Conference 2018 (2018)
Li, M., Wang, B., Jiang, J.: Siamese pre-trained transformer encoder for knowledge base completion. Neural Process. Lett. (2021)
Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. ArXiv abs/1907.11692 (2019)
Mahdisoltani, F., Biega, J.A., Suchanek, F.M.: Yago3: a knowledge base from multilingual wikipedias. In: Conference on Innovative Data Systems Research (2015)
Mavromatis, C., et al.: Tempoqr: temporal question reasoning over knowledge graphs. In: AAAI Conference on Artificial Intelligence (2021)
Nguyen, D.Q., Nguyen, D.Q., Nguyen, T.D., Phung, D.Q.: A convolutional neural network-based model for knowledge base completion and its application to search personalization. Semantic Web (2019)
Ostendorff, M., Bourgonje, P., Berger, M., Schneider, J.M., Rehm, G., Gipp, B.: Enriching bert with knowledge graph embeddings for document classification. ArXiv abs/1909.08402 (2019)
Pedregosa, F., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. (2011)
Petroni, F., et al.: Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2019)
Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using Siamese Bert-networks. In: Conference on Empirical Methods in Natural Language Processing (EMNLP) (2019)
Ren, H., Hu, W., Leskovec, J.: Query2box: reasoning over knowledge graphs in vector space using box embeddings. In: International Conference on Learning Representations (2020)
Rezatofighi, S.H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I.D., Savarese, S.: Generalized intersection over union: a metric and a loss for bounding box regression. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
Rogers, A., Kovaleva, O., Rumshisky, A.: A primer in bertology: what we know about how bert works. Trans. Assoc. Comput. Linguist. (2020)
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embeddings for simple link prediction. In: International Conference on Machine Learning (2016)
Vaswani, A., et al.: Attention is all you need. In: Neural Information Processing Systems (2017)
Wang, L., Zhao, W., Wei, Z., Liu, J.: Simkgc: simple contrastive knowledge graph completion with pre-trained language models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (2022)
Weikum, G., Dong, L., Razniewski, S., Suchanek, F.M.: Machine knowledge: creation and curation of comprehensive knowledge bases. Found. Trends Databases (2020)
Xie, R., Liu, Z., Jia, J., Luan, H., Sun, M.: Representation learning of knowledge graphs with entity descriptions. In: AAAI Conference on Artificial Intelligence (2016)
Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. In: International Conference on Learning Representations (ICLR) (2015)
Yao, L., Mao, C., Luo, Y.: KG-BERT: bert for knowledge graph completion. ArXiv abs/1909.03193 (2019)
Zhang, X., et al.: Temporal context-aware representation learning for question routing. In: Proceedings of the 13th International Conference on Web Search and Data Mining (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
7Appendix
7Appendix
1.1 7.1Effect of Number of Negative Samples
We conduct an empirical study to see how sampling types discussed in Sect. 3 affect the performance of TEMT\(_{ND}\). We analyze different number of entity-corrupted and time-corrupted negative samples on YAGO11k and ind-YAGO11k datasets. The results are reported in Table 6. We perform the same experiments on Wikidata12k and ind-Wikidata12k as well, however, we do not include their results here, for the sake of brevity.
The results in Table 6 show that the entity-corrupted negative sampling performs worse than the time-corrupted negative sampling for both datasets. Since the time interval prediction also requires the model to distinguish facts with different time points, this difference is expected. Moreover, in time-corrupted cases, the number of negative samples does not result in marginal changes in gaeIOU@1 metric, which is the most stringent metric.
Time prediction evaluation terms
1.2 7.2Evaluation Terminology
In this section, we illustrate the evaluation terms employed within our interval metrics. In Fig. 3, we demonstrate two scenarios when the predicted interval overlaps with the gold interval (top figure) and when it does not (bottom figure). In the case of the former, given \(I_p=[2002, 2006]\) and \(I_g=[2004, 2008]\), we get the following: the \(hull(I_p, I_g)=[2002, 2008]\), and the \(overlap(I_p, I_g)=[2004, 2006]\). In the latter case, given \(I_p=[2001, 2003]\) and \(I_g=[2005, 2008]\), then the \(hull(I_p, I_g)=[2001, 2008]\) and \(|gap(I_p, I_g)|=3\).
1.3 7.3Variance Analysis
Since our experimental results are averaged over the test triples, we report the variance values in Table 7. The results across the datasets and the variants illustrate the effectiveness of our model.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Islakoglu, D.S., Chekol, M.W., Velegrakis, Y. (2024). Leveraging Pre-trained Language Models for Time Interval Prediction in Text-Enhanced Temporal Knowledge Graphs. In: Meroño Peñuela, A., et al. The Semantic Web. ESWC 2024. Lecture Notes in Computer Science, vol 14664. Springer, Cham. https://doi.org/10.1007/978-3-031-60626-7_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-60626-7_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-60625-0
Online ISBN: 978-3-031-60626-7
eBook Packages: Computer ScienceComputer Science (R0)