More Web Proxy on the site http://driver.im/

research-article

MEGA: Meta-Graph Augmented Pre-Training Model for Knowledge Graph Completion

Authors:

Xiaoling ZhuAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data, Volume 18, Issue 1

Article No.: 30, Pages 1 - 24

https://doi.org/10.1145/3617379

Published: 16 October 2023 Publication History

Abstract

Nowadays, a large number of Knowledge Graph Completion (KGC) methods have been proposed by using embedding based manners, to overcome the incompleteness problem faced with knowledge graph (KG). One important recent innovation in Natural Language Processing (NLP) domain is the employ of deep neural models that make the most of pre-training, culminating in BERT, the most popular example of this line of approaches today. Recently, a series of new KGC methods introducing a pre-trained language model, such as KG-BERT, have been developed and released compelling performance. However, previous pre-training based KGC methods usually train the model by using simple training task and only utilize one-hop relational signals in KG, which leads that they cannot model high-order semantic contexts and multi-hop complex relatedness. To overcome this problem, this article presents a novel pre-training framework for KGC task, which especially consists of both one-hop relation level task (low-order) and multi-hop meta-graph level task (high-order). Hence, the proposed method can capture not only the elaborate sub-graph structure but also the subtle semantic information on the given KG. The empirical results show the efficiency of the proposed method on the widely used real-world datasets.

References

[1]

Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. TuckER: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 5185–5194.

[2]

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th International Conference on Neural Information Processing Systems. 2787–2795.

[3]

Chandrahas and Partha P. Talukdar. 2021. OKGIT: Open knowledge graph link prediction with implicit types. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21). 2546–2559.

[4]

Bonggeun Choi, Daesik Jang, and Youngjoong Ko. 2021. MEM-KGC: Masked entity model for knowledge graph completion with pre-trained language model. IEEE Access 9 (2021), 132025–132032. https://ieeexplore.ieee.org/document/9540703/

[5]

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2D knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence.

[6]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-HLT.

[7]

Yuan Fang, Wenqing Lin, Vincent Wenchen Zheng, Min Wu, Kevin Chen-Chuan Chang, and Xiaoli Li. 2016. Semantic proximity search on graphs with metagraph-based learning. In Proceedings of the 2016 IEEE 32nd International Conference on Data Engineering. 277–288.

[8]

Jianlong Fu, Jinqiao Wang, Yong Rui, Xin-Jing Wang, Tao Mei, and Hanqing Lu. 2015. Image tag refinement with view-dependent concept representations. IEEE Transactions on Circuits and Systems for Video Technology 25, 8 (2015), 1409–1422.

[9]

Haipeng Gao, Kun Yang, Yuxue Yang, Rufai Yusuf Zakari, Jim Wilson Owusu, and Ke Qin. 2021. QuatDE: Dynamic quaternion embedding for knowledge graph completion. arXiv:2105.09002. Retrieved from https://arxiv.org/abs/2105.09002

[10]

Lingbing Guo, Weiqing Wang, Zequn Sun, Chenghao Liu, and Wei Hu. 2020. Decentralized knowledge graph representation learning. arXiv:2010.08114. Retrieved from https://arxiv.org/abs/2010.08114

[11]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9726–9735.

[12]

Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In Proceedings of the Web Conference 2020.

Digital Library

[13]

Heyan Huang, Yashen Wang, Chong Feng, Zhirun Liu, and Qiang Zhou. 2018. Leveraging conceptualization for short-text embedding. IEEE Transactions on Knowledge and Data Engineering 30, 7 (2018), 1282–1295.

[14]

Zhipeng Huang, Yudian Zheng, Reynold Cheng, Yizhou Sun, Nikos Mamoulis, and Xiang Li. 2016. Meta structure: Computing relevance in large heterogeneous information networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Digital Library

[15]

Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jian Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics.

[16]

Shaoxiong Ji, Shirui Pan, E. Cambria, P. Marttinen, and Philip S. Yu. 2021. A survey on knowledge graphs: Representation, acquisition and applications. IEEE Transactions on Neural Networks and Learning Systems 33, 2 (2021), 494–514.

[17]

Xunqiang Jiang, Yuanfu Lu, Yuan Fang, and Chuan Shi. 2021. Contrastive pre-training of GNNs on heterogeneous graphs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management.

[18]

Sanjay Kamath, Brigitte Grau, and Yue Ma. 2019. How to pre-train your model? Comparison of different pre-training models for biomedical question answering. In Proceedings of Machine Learning and Knowledge Discovery in Databases International Workshops of (ECML-PKDD’19). 646–660.

[19]

Bosung Kim, Taesuk Hong, Youngjoong Ko, and Jungyun Seo. 2020. Multi-task learning for knowledge graph completion with pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics.

[20]

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence. 2181–2187.

[21]

Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juan-Zi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. 2022. Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. In Findings of the Association for Computational Linguistics: (ACL’22). 3570–3581.

[22]

Shiheng Ma, Jianhui Ding, Weijia Jia, Kun Wang, and Minyi Guo. 2017. TransT: Type-based multiple embedding representations for knowledge graph completion. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. 717–733.

[23]

Guanglin Niu, Bo Li, Yongfei Zhang, and Shi Pu. 2022. CAKE: A scalable commonsense-aware framework for multi-view knowledge graph completion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.

[24]

Guanglin Niu, Bo Li, Yongfei Zhang, Shiliang Pu, and Jingyan Li. 2020. AutoETER: Automated entity type representation for knowledge graph embedding. In Findings of the Association for Computational Linguistics: (EMNLP’20). 1172–1181.

[25]

Jin-woo Park, Seung-won Hwang, and Haixun Wang. 2016. Fine-grained semantic conceptualization of FrameNet. In Proceedings of the AAAI Conference on Artificial Intelligence. 2638–2644.

[26]

Weizhen Qi, Yeyun Gong, Yu Yan, Can Xu, Bolun Yao, Bartuer Zhou, Biao Cheng, Daxin Jiang, Jiusheng Chen, Ruofei Zhang, Houqiang Li, and Nan Duan. 2021. ProphetNet-X: Large-scale pre-training models for English, Chinese, multi-lingual, dialog, and code generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics.

[27]

Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training. https://www.mikecaptain.com/resources/pdf/GPT-1.pdf

[28]

Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and Philip S. Yu. 2017. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering 29, 1 (2017), 17–37.

Digital Library

[29]

Richard Socher, Danqi Chen, Christopher D. Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of the 26th International Conference on Neural Information Processing Systems. 926–934.

[30]

Yangqiu Song, Haixun Wang, Zhongyuan Wang, Hongsong Li, and Weizhu Chen. 2011. Short text conceptualization using a probabilistic knowledgebase. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence. Vol. 3, 2330–2336.

[31]

Yangqiu Song, Haixun Wang, Zhongyuan Wang, Hongsong Li, and Weizhu Chen. 2011. Short text conceptualization using a probabilistic knowledgebase. In Proceedings of the International Joint Conference on Artificial Intelligence. 2330–2336.

[32]

Yangqiu Song, Shusen Wang, and Haixun Wang. 2015. Open domain short text conceptualization: A generative + descriptive modeling approach. In Proceedings of the 24th International Conference on Artificial Intelligence.

[33]

Yangqiu Song, Shusen Wang, and Haixun Wang. 2015. Open domain short text conceptualization: A generative + descriptive modeling approach. In Proceedings of the International Conference on Artificial Intelligence. 3820–3826.

[34]

Zhiqing Sun, Zhihong Deng, Jian-Yun Nie, and Jian Tang. 2019. RotatE: Knowledge graph embedding by relational rotation in complex space. arXiv:1902.10197. Retrieved from https://arxiv.org/abs/1902.10197

[35]

Jinhui Tang, Xiangbo Shu, Zechao Li, Yu-Gang Jiang, and Qi Tian. 2019. Social anchor-unit graph regularized tensor completion for large-scale image retagging. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 8 (2019), 2027–2034.

[36]

Jinhui Tang, Xiangbo Shu, Guo-Jun Qi, Zechao Li, Meng Wang, Shuicheng Yan, and Ramesh C. Jain. 2017. Tri-clustered tensor completion for social-aware image tag refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 8 (2017), 1662–1674.

[37]

Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality.

[38]

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv:1807.03748. Retrieved from https://arxiv.org/abs/1807.03748

[39]

Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, and Yi Chang. 2021. Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference 2021.

Digital Library

[40]

Fang Wang, Zhongyuan Wang, Zhoujun Li, and Ji Rong Wen. 2014. Concept-based short text classification and ranking. In Proceedings of the ACM International Conference. 1069–1078.

Digital Library

[41]

Lili Wang, Chongyang Gao, Chenghan Huang, Ruibo Liu, Weicheng Ma, and Soroush Vosoughi. 2021. Embedding heterogeneous networks into hyperbolic space without meta-path. In Proceedings of the AAAI Conference on Artificial Intelligence.

[42]

Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.

[43]

Q. Wang, Zhendong Mao, B. Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering 29, 12 (2017), 2724–2743.

[44]

Xiao Wang, Yuanfu Lu, Chuan Shi, Ruijia Wang, Peng Cui, and Shuai Mou. 2022. Dynamic heterogeneous information network embedding with meta-path based proximity. IEEE Transactions on Knowledge and Data Engineering 34, 3 (2022), 1117–1132.

[45]

Yashen Wang, Heyan Huang, and Chong Feng. 2017. Query expansion based on a feedback concept model for microblog retrieval. In Proceedings of the International Conference on World Wide Web. 559–568.

Digital Library

[46]

Yashen Wang, Heyan Huang, Chong Feng, Qiang Zhou, Jiahui Gu, and Xiong Gao. 2016. CSE: Conceptual sentence embeddings based on attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 505–515.

[47]

Yashen Wang, Yifeng Liu, Huanhuan Zhang, and Haiyong Xie. 2019. Leveraging lexical semantic information for learning concept-based multiple embedding representations for knowledge graph completion. In Proceedings of the Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data.

Digital Library

[48]

Yashen Wang, Xiaoye Ouyang, Xiaoling Zhu, and Huanhuan Zhang. 2022. Concept commons enhanced knowledge graph representation. In Proceedings of the 15th International Conference on Knowledge Science, Engineering and Management. Gérard Memmi, Baijian Yang, Linghe Kong, Tianwei Zhang, and Meikang Qiu (Eds.), Lecture Notes in Computer Science, Vol. 13368, Springer, 413–424. DOI:

Digital Library

[49]

Yashen Wang, Zhaoyu Wang, Huanhuan Zhang, and Zhirun Liu. 2022. Microblog retrieval based on concept-enhanced pre-training model. ACM Transactions on Knowledge Discovery from Data 17, 3 (2022), 41:1–41:32.

[50]

Yashen Wang, He yan Huang, and Chong Feng. 2021. Query expansion with local conceptual word embeddings in microblog retrieval. IEEE Transactions on Knowledge and Data Engineering 33, 4 (2021), 1737–1749.

[51]

Yashen Wang, Huanhuan Zhang, Yifeng Li, and Haiyong Xie. 2019. Simplified representation learning model based on parameter-sharing for knowledge graph completion. In Proceedings of the 25th China Conference on Information Retrieval.

Digital Library

[52]

Yashen Wang, Huanhuan Zhang, Zhirun Liu, and Qiang Zhou. 2021. Hierarchical concept-driven language model. ACM Transactions on Knowledge Discovery from Data 15, 6 (2021), 1–22.

[53]

Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence. 1112–1119.

[54]

Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, and Ji-Rong Wen. 2015. Query understanding through knowledge-based conceptualization. In Proceedings of the 24th International Conference on Artificial Intelligence.

Digital Library

[55]

Zhongyuan Wang, Kejun Zhao, Haixun Wang, Xiaofeng Meng, and Ji Rong Wen. 2015. Query understanding through knowledge-based conceptualization. In Proceedings of the International Conference on Artificial Intelligence. 3264–3270.

Digital Library

[56]

Lei Wu, Xiansheng Hua, Nenghai Yu, Wei-Ying Ma, and Shipeng Li. 2012. Flickr distance: A relationship measure for visual concepts. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 5 (2012), 863–875.

[57]

Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Q. Zhu. 2012. Probase: A probabilistic taxonomy for text understanding. In Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. 481–492.

Digital Library

[58]

Han Xiao, Minlie Huang, Lian Meng, and Xiaoyan Zhu. 2017. SSP: Semantic space projection for knowledge graph embedding with text descriptions. In Proceedings of the AAAI Conference on Artificial Intelligence.

[59]

Han Xiao, Minlie Huang, and Xiaoyan Zhu. 2016. TransG : A generative model for knowledge graph embedding. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. 2316–2325.

[60]

Fenfang Xie, Angyu Zheng, Liang Chen, and Zibin Zheng. 2021. Attentive meta-graph embedding for item recommendation in heterogeneous information networks. Knowledge-Based Systems 211 (2021), 106524. https://www.sciencedirect.com/science/article/abs/pii/S0950705120306535?via%3Dihub

[61]

Ruobing Xie, Zhiyuan Liu, J. J. Jia, Huanbo Luan, and Maosong Sun. 2016. Representation learning of knowledge graphs with entity descriptions. In Proceedings of the AAAI Conference on Artificial Intelligence.

[62]

Bishan Yang, Wen tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv:1412.6575. Retrieved from https://arxiv.org/abs/1412.6575

[63]

Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. arXiv:1909.03193. Retrieved from https://arxiv.org/abs/1909.03193

[64]

Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2019. Learning hierarchy-aware knowledge graph embeddings for link prediction. In Proceedings of the AAAI Conference on Artificial Intelligence.

[65]

Huan Zhao, Quanming Yao, Jianda Li, Yangqiu Song, and Dik Lun Lee. 2017. Meta-graph based recommendation fusion over heterogeneous information networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.

Digital Library

Cited By

Mao QLi ZLiu QWu LZhang HChen E(2024)Promoting Machine Abilities of Discovering and Utilizing Knowledge in a Unified Zero-Shot Learning ParadigmACM Transactions on Knowledge Discovery from Data10.1145/370044419:1(1-26)Online publication date: 30-Nov-2024
https://doi.org/10.1145/3700444
Sun KJiang HHu YYin B(2024)Incorporating Multi-Level Sampling with Adaptive Aggregation for Inductive Knowledge Graph CompletionACM Transactions on Knowledge Discovery from Data10.1145/364482218:5(1-16)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644822
Wang YZhu XChen TZhang Y(2024)An Aggregation Procedure Optimization Method by Leveraging Neighboring Prompt for GCN-based Knowledge Graph Completion Model2024 IEEE 9th International Conference on Data Science in Cyberspace (DSC)10.1109/DSC63484.2024.00042(263-270)Online publication date: 23-Aug-2024
https://doi.org/10.1109/DSC63484.2024.00042
Show More Cited By

Recommendations

Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion
Knowledge graphs often suffer from incompleteness, and knowledge graph completion (KGC) aims at inferring the missing triplets through knowledge graph embedding from known factual triplets. However, most existing knowledge graph embedding methods only use ...
Multi-Concept Representation Learning for Knowledge Graph Completion
Knowledge Graph Completion (KGC) aims at inferring missing entities or relations by embedding them in a low-dimensional space. However, most existing KGC methods generally fail to handle the complex concepts hidden in triplets, so the learned embeddings ...
KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion
WWW '23: Proceedings of the ACM Web Conference 2023

Knowledge Graph Embeddings (KGE) aim to map entities and relations to low dimensional spaces and have become the de-facto standard for knowledge graph completion. Most existing KGE methods suffer from the sparsity challenge, where it is harder to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 18, Issue 1

January 2024

854 pages

EISSN:1556-472X

DOI:10.1145/3613504

Editor:
Charu Aggarwal
IBM T. J. Watson Research, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2023

Online AM: 25 August 2023

Accepted: 28 July 2023

Revised: 03 April 2023

Received: 02 December 2022

Published in TKDD Volume 18, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
799
Total Downloads

Downloads (Last 12 months)360
Downloads (Last 6 weeks)14

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mao QLi ZLiu QWu LZhang HChen E(2024)Promoting Machine Abilities of Discovering and Utilizing Knowledge in a Unified Zero-Shot Learning ParadigmACM Transactions on Knowledge Discovery from Data10.1145/370044419:1(1-26)Online publication date: 30-Nov-2024
https://doi.org/10.1145/3700444
Sun KJiang HHu YYin B(2024)Incorporating Multi-Level Sampling with Adaptive Aggregation for Inductive Knowledge Graph CompletionACM Transactions on Knowledge Discovery from Data10.1145/364482218:5(1-16)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3644822
Wang YZhu XChen TZhang Y(2024)An Aggregation Procedure Optimization Method by Leveraging Neighboring Prompt for GCN-based Knowledge Graph Completion Model2024 IEEE 9th International Conference on Data Science in Cyberspace (DSC)10.1109/DSC63484.2024.00042(263-270)Online publication date: 23-Aug-2024
https://doi.org/10.1109/DSC63484.2024.00042
Si JOuyang XZhu XZhang Y(2024)A Relation Semantic Enhancement Method for Large Language Model Based Knowledge Graph Completion2024 IEEE 9th International Conference on Data Science in Cyberspace (DSC)10.1109/DSC63484.2024.00035(209-215)Online publication date: 23-Aug-2024
https://doi.org/10.1109/DSC63484.2024.00035
Sun KJiang HHu YYin B(2024)Generating Graph-Based Rules for Enhancing Logical ReasoningAdvanced Intelligent Computing Technology and Applications10.1007/978-981-97-5615-5_12(143-156)Online publication date: 5-Aug-2024
https://dl.acm.org/doi/10.1007/978-981-97-5615-5_12

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents