More Web Proxy on the site http://driver.im/

research-article

Open access

PSNEA: Pseudo-Siamese Network for Entity Alignment between Multi-modal Knowledge Graphs

Authors:

Yangbangyan Jiang,

Qingming HuangAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 3489 - 3497

https://doi.org/10.1145/3581783.3611778

Published: 27 October 2023 Publication History

Abstract

Multi-modal entity alignment aims to identify entities that refer to the same concept in the real world across a plethora of multi-modal knowledge graphs (MMKGs). Most existing methods focus on reducing the embedding differences between multiple modalities while neglecting the following challenges: 1) cannot handle the heterogeneity across graphs, 2) suffer from the scarcity of pre-aligned data (a.k.a. initial seeds). To tackle these issues, we propose a Pseudo-Siamese Network for multi-modal Entity Alignment (PSNEA). It consists of two modules to extract various information and generate holistic embeddings. Specifically, the first module PSN is designed with two parallel branches to learn the representations for different MMKGs, thus effectively bridging the graph heterogeneity. On top of this, we introduce an Incremental Alignment Pool (IAP) to alleviate the scarcity of initial seeds by labeling likely alignment. IAP avoids error-prone by data swapping and sample re-weighting strategies. To the best of our knowledge, PSNEA is the first model that tackles graph heterogeneity and scarcity of initial seeds in one unified framework. The extensive experiments demonstrate that our model achieves the best performance on both cross-lingual and cross-graph datasets. The source code is available at https://github.com/idrfer/psn4ea.

References

[1]

Abney, S. P. Understanding the yarowsky algorithm. Comput. Linguistics 30, 3 (2004), 365--395.

Digital Library

[2]

Awasthi, P., Dikkala, N., and Kamath, P. Do more negative samples necessarily hurt in contrastive learning? In International Conference on Machine Learning (2022), vol. 162 of Proceedings of Machine Learning Research, PMLR, pp. 1101--1116.

[3]

Bordes, A., Glorot, X., Weston, J., and Bengio, Y. A semantic matching energy function for learning with multi-relational data - application to word-sense disambiguation. Mach. Learn. 94, 2 (2014), 233--259.

Digital Library

[4]

Bordes, A., Usunier, N., García-Durán, A., Weston, J., and Yakhnenko, O. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013 (2013), pp. 2787--2795.

[5]

Cao, Y., Liu, Z., Li, C., Liu, Z., Li, J., and Chua, T. Multi-channel graph neural network for entity alignment. In Proceedings of the 57th Conference of the Association for Computational Linguistics (2019), Association for Computational Linguistics, pp. 1452--1461.

[6]

Cao, Z., Xu, Q., Yang, Z., Cao, X., and Huang, Q. Dual quaternion knowledge graph embeddings. In Thirty-Fifth AAAI Conference on Artificial Intelligence (2021), AAAI Press, pp. 6894--6902.

[7]

Cao, Z., Xu, Q., Yang, Z., He, Y., Cao, X., and Huang, Q. OTKGE: multi-modal knowledge graph embeddings via optimal transport. In NeurIPS (2022).

[8]

Chai, J., and Shi, G. Module: Module embedding for knowledge graphs. CoRR abs/2203.04702 (2022).

[9]

Chen, L., Li, Z., Wang, Y., Xu, T., Wang, Z., and Chen, E. MMEA: entity alignment for multi-modal knowledge graph. In Knowledge Science, Engineering and Management - 13th International Conference (2020), vol. 12274 of Lecture Notes in Computer Science, Springer, pp. 134--147.

Digital Library

[10]

Chen, L., Li, Z., Xu, T., Wu, H., Wang, Z., Yuan, N. J., and Chen, E. Multi-modal siamese network for entity alignment. In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (2022), ACM, pp. 118--126.

Digital Library

[11]

Chen, M., Tian, Y., Chang, K., Skiena, S., and Zaniolo, C. Co-training embeddings of knowledge graphs and entity descriptions for cross-lingual entity alignment. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (2018), ijcai.org, pp. 3998--4004.

[12]

Chen, M., Tian, Y., Yang, M., and Zaniolo, C. Multilingual knowledge graph embeddings for cross-lingual knowledge alignment. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (2017), ijcai.org, pp. 1511--1517.

[13]

Gunel, B., Du, J., Conneau, A., and Stoyanov, V. Supervised contrastive learning for pre-trained language model fine-tuning. In 9th International Conference on Learning Representations (2021), OpenReview.net.

[14]

Hu, W., Zhang, Q., Sun, Z., and Huang, J. Multike: a multi-view knowledge graph embedding framework for entity alignment. In Proceedings of the 14th International Workshop on Ontology Matching co-located with the 18th International Semantic Web Conference (2019), vol. 2536 of CEUR Workshop Proceedings, CEUR-WS.org, pp. 189--190.

[15]

Ji, G., He, S., Xu, L., Liu, K., and Zhao, J. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (2015), The Association for Computer Linguistics, pp. 687--696.

[16]

Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P. N., Hellmann, S., Morsey, M., van Kleef, P., Auer, S., and Bizer, C. Dbpedia - A large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web 6, 2 (2015), 167--195.

[17]

Lin, Y., Liu, Z., Sun, M., Liu, Y., and Zhu, X. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015), AAAI Press, pp. 2181--2187.

Digital Library

[18]

Lin, Z., Zhang, Z., Wang, M., Shi, Y., Wu, X., and Zheng, Y. Multi-modal contrastive representation learning for entity alignment. In Proceedings of the 29th International Conference on Computational Linguistics (2022), International Committee on Computational Linguistics, pp. 2572--2584.

[19]

Liu, F., Chen, M., Roth, D., and Collier, N. Visual pivoting for (unsupervised) entity alignment. In Thirty-Fifth AAAI Conference on Artificial Intelligence (2021), AAAI Press, pp. 4257--4266.

[20]

Liu, Y., Li, H., García-Durán, A., Niepert, M., Oñoro-Rubio, D., and Rosenblum, D. S. MMKG: multi-modal knowledge graphs. In The Semantic Web - 16th International Conference (2019), vol. 11503 of Lecture Notes in Computer Science, Springer, pp. 459--474.

Digital Library

[21]

Lu, X., Wang, L., Jiang, Z., He, S., and Liu, S. MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning. Appl. Intell. 52, 7 (2022), 7480--7497.

Digital Library

[22]

Nickel, M., Tresp, V., and Kriegel, H. A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning (2011), Omnipress, pp. 809--816.

Digital Library

[23]

Oñoro-Rubio, D., Niepert, M., García-Durán, A., Gonzalez-Sanchez, R., and López-Sastre, R. J. Answering visual-relational queries in web-extracted knowledge graphs. In 1st Conference on Automated Knowledge Base Construction (2019).

[24]

Radhakrishnan, P., Talukdar, P. P., and Varma, V. ELDEN: improved entity linking using densified knowledge graphs. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2018), Association for Computational Linguistics, pp. 1844--1853.

[25]

Sergieh, H. M., Botschen, T., Gurevych, I., and Roth, S. A multimodal translation-based approach for knowledge graph representation learning. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (2018), Association for Computational Linguistics, pp. 225--234.

[26]

Socher, R., Chen, D., Manning, C. D., and Ng, A. Y. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013 (2013), pp. 926--934.

[27]

Sun, R., Cao, X., Zhao, Y., Wan, J., Zhou, K., Zhang, F., Wang, Z., and Zheng, K. Multi-modal knowledge graphs for recommender systems. In The 29th ACM International Conference on Information and Knowledge Management (2020), ACM, pp. 1405--1414.

Digital Library

[28]

Sun, Z., Deng, Z., Nie, J., and Tang, J. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th International Conference on Learning Representations (2019), OpenReview.net.

[29]

Sun, Z., Hu, W., and Li, C. Cross-lingual entity alignment via joint attribute-preserving embedding. In The Semantic Web - ISWC 2017 - 16th International Semantic Web Conference (2017), vol. 10587 of Lecture Notes in Computer Science, Springer, pp. 628--644.

Digital Library

[30]

Sun, Z., Hu, W., and Li, C. Cross-lingual entity alignment via joint attribute-preserving embedding. In The Semantic Web - ISWC 2017 - 16th International Semantic Web Conference (2017), vol. 10587 of Lecture Notes in Computer Science, Springer, pp. 628--644.

Digital Library

[31]

Sun, Z., Hu, W., Zhang, Q., and Qu, Y. Bootstrapping entity alignment with knowledge graph embedding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (2018), ijcai.org, pp. 4396--4402.

[32]

Sun, Z., Hu, W., Zhang, Q., and Qu, Y. Bootstrapping entity alignment with knowledge graph embedding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (2018), ijcai.org, pp. 4396--4402.

[33]

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016), IEEE Computer Society, pp. 2818--2826.

[34]

Trisedya, B. D., Qi, J., and Zhang, R. Entity alignment between knowledge graphs using attribute embeddings. In The Thirty-Third AAAI Conference on Artificial Intelligence (2019), AAAI Press, pp. 297--304.

Digital Library

[35]

Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., and Bouchard, G. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning (2016), vol. 48 of JMLR Workshop and Conference Proceedings, JMLR.org, pp. 2071--2080.

[36]

Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. Graph attention networks. In 6th International Conference on Learning Representations (2018), OpenReview.net.

[37]

Wang, Z., Lv, Q., Lan, X., and Zhang, Y. Cross-lingual knowledge graph alignment via graph convolutional networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018), Association for Computational Linguistics, pp. 349--357.

[38]

Wang, Z., Zhang, J., Feng, J., and Chen, Z. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (2014), AAAI Press, pp. 1112--1119.

Digital Library

[39]

Xie, R., Liu, Z., Luan, H., and Sun, M. Image-embodied knowledge representation learning. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (2017), ijcai.org, pp. 3140--3146.

[40]

Yang, B., Yih, W., He, X., Gao, J., and Deng, L. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations (2015).

[41]

Yang, Z., Tang, J., and Cohen, W. W. Multi-modal bayesian embeddings for learning social knowledge graphs. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (2016), IJCAI/AAAI Press, pp. 2287--2293.

Digital Library

[42]

Zhang, Z., Liu, H., Chen, J., Chen, X., Liu, B., Xiang, Y., and Zheng, Y. An industry evaluation of embedding-based entity alignment. In Proceedings of the 28th International Conference on Computational Linguistics (2020), International Committee on Computational Linguistics, pp. 179--189.

[43]

Zhu, H., Xie, R., Liu, Z., and Sun, M. Iterative entity alignment via joint knowl-edge embeddings. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (2017), ijcai.org, pp. 4258--4264.

[44]

Zhu, Y., Zhang, C., Ré, C., and Fei-Fei, L. Building a large-scale multi-modal knowledge base system for answering visual queries. arXiv preprint arXiv:1507.05670 (2015).

Cited By

Zhou QChen WZhang LLiu AFang JZhao L(2025)ADMH-ER: Adaptive Denoising Multi-Modal Hybrid for Entity ResolutionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2025.352662337:3(1049-1063)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2025.3526623
Zhao YZhang YSui XCai X(2025) ME A: A Multimodal Entity Entailment framework for multimodal Entity Alignment Information Processing & Management10.1016/j.ipm.2024.10395162:1(103951)Online publication date: Jan-2025
https://doi.org/10.1016/j.ipm.2024.103951
Liu T(2024)Multi-modal Knowledge Graph Completion: A SurveyProceedings of the International Conference on Image Processing, Machine Learning and Pattern Recognition10.1145/3700906.3700925(116-121)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3700906.3700925
Show More Cited By

Index Terms

PSNEA: Pseudo-Siamese Network for Entity Alignment between Multi-modal Knowledge Graphs
1. Information systems
  1. Information systems applications
    1. Data mining
    2. Multimedia information systems

Recommendations

Multi-modal Siamese Network for Entity Alignment
KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

The booming of multi-modal knowledge graphs (MMKGs) has raised the imperative demand for multi-modal entity alignment techniques, which facilitate the integration of multiple MMKGs from separate data sources. Unfortunately, prior arts harness multi-modal ...
MMEA: Entity Alignment for Multi-modal Knowledge Graph
Knowledge Science, Engineering and Management
Abstract
Entity alignment plays an essential role in the knowledge graph (KG) integration. Though large efforts have been made on exploring the association of relational embeddings between different knowledge graphs, they may fail to effectively describe ...
Multi-modal Graph Convolutional Network for Knowledge Graph Entity Alignment
Web and Big Data
Abstract
Entity Alignment (EA) plays a crucial role in the integration of multiple knowledge graphs (KGs). With the blooming of KGs, the auxiliary multi-modal data, such as attributions and images, are widely used to enhance alignment performance. However, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities
Youth Inno- vation Promotion Association CAS
Strategic Priority Research Program of Chinese Academy of Sciences
National Key R&D Program of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
849
Total Downloads

Downloads (Last 12 months)596
Downloads (Last 6 weeks)87

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou QChen WZhang LLiu AFang JZhao L(2025)ADMH-ER: Adaptive Denoising Multi-Modal Hybrid for Entity ResolutionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2025.352662337:3(1049-1063)Online publication date: Mar-2025
https://doi.org/10.1109/TKDE.2025.3526623
Zhao YZhang YSui XCai X(2025) ME A: A Multimodal Entity Entailment framework for multimodal Entity Alignment Information Processing & Management10.1016/j.ipm.2024.10395162:1(103951)Online publication date: Jan-2025
https://doi.org/10.1016/j.ipm.2024.103951
Liu T(2024)Multi-modal Knowledge Graph Completion: A SurveyProceedings of the International Conference on Image Processing, Machine Learning and Pattern Recognition10.1145/3700906.3700925(116-121)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3700906.3700925
Wang YXu QJiang YDai SHuang QCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Regularized Contrastive Partial Multi-view Outlier DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681125(8711-8720)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681125
Junwei HXu QJiang YWang ZSun YHuang QCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution DetectionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681118(1544-1553)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681118
Su TSheng JWang SZhang XXu HLiu TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity AlignmentProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680954(4436-4445)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680954
Xu YLi YXu MZhu ZZhao Y(2024)HKA: A Hierarchical Knowledge Alignment Framework for Multimodal Knowledge Graph CompletionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366428820:8(1-19)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3664288
Zhang SZeng WTan ZZhao XXiao WSerra ESpezzano F(2024)M3: A Multi-Image Multi-Modal Entity Alignment DatasetProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679126(5415-5419)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679126
Zhu BWang RWang JShao FWang K(2024)A survey: knowledge graph entity alignment research based on graph embeddingArtificial Intelligence Review10.1007/s10462-024-10866-457:9Online publication date: 3-Aug-2024
https://doi.org/10.1007/s10462-024-10866-4

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten