More Web Proxy on the site http://driver.im/

research-article

Free access

Contrastive Cross-scale Graph Knowledge Synergy

Authors:

Irwin KingAuthors Info & Claims

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 3422 - 3433

https://doi.org/10.1145/3580305.3599286

Published: 04 August 2023 Publication History

Abstract

Graph representation learning via Contrastive Learning (GCL) has drawn considerable attention recently. Efforts are mainly focused on gathering more global information via contrasting on a single high-level graph view, which, however, underestimates the inherent complex and hierarchical properties in many real-world networks, leading to sub-optimal embeddings. To incorporate these properties of a complex graph, we propose Cross-Scale Contrastive Graph Knowledge Synergy (CGKS), a generic feature learning framework, to advance graph contrastive learning with enhanced generalization ability and the awareness of latent anatomies. Specifically, to maintain the hierarchical information, we create a so-call graph pyramid (GP) consisting of coarse-grained graph views. Each graph view is obtained via the careful design topology-aware graph coarsening layer that extends the Laplacian Eigenmaps with negative sampling. To promote cross-scale information sharing and knowledge interactions among GP, we propose a novel joint optimization formula that contains a pairwise contrastive loss between any two coarse-grained graph views. This synergy loss not only promotes knowledge sharing that yields informative representations, but also stabilizes the training process. Experiments on various downstream tasks demonstrate the substantial improvements of the proposed method over its counterparts.

Supplementary Material

MP4 File (rtfp1118-2min-promo.mp4)

Promotional video.

Download
4.68 MB

MP4 File (rtfp1118-20min-video.mp4)

Presentation video

Download
40.70 MB

References

[1]

Amir Hosein Khas Ahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. 2020. Memory-Based Graph Networks. In Proc. of ICLR.

[2]

Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. 2020. Spectral Clustering with Graph Neural Networks for Graph Pooling. In Proc. of ICML.

[3]

Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. 2017. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine.

[4]

Yankai Chen, Yixiang Fang, Yifei Zhang, and Irwin King. 2023 a. Bipartite Graph Convolutional Hashing for Effective and Efficient Top-N Search in Hamming Space. In Proceedings of the ACM Web Conference 2023 (WebConf).

Digital Library

[5]

Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. 2022a. Learning binarized graph representations with multi-faceted quantization reinforcement for top-k recommendation. In Proc. of KDD.

Digital Library

[6]

Yankai Chen, Menglin Yang, Yingxue Zhang, Mengchen Zhao, Ziqiao Meng, Jianye Hao, and Irwin King. 2022c. Modeling scale-free graphs with hyperbolic geometry for knowledge-aware recommendation. In Proc. of WSDM.

Digital Library

[7]

Yankai Chen, Yaming Yang, Yujing Wang, Jing Bai, Xiangchen Song, and Irwin King. 2022b. Attentive knowledge-aware graph convolutional networks with collaborative guidance for personalized recommendation. In 2022 IEEE 38th International Conference on Data Engineering (ICDE).

[8]

Yankai Chen, Yifei Zhang, Huifeng Guo, Ruiming Tang, and Irwin King. 2022d. An Effective Post-training Embedding Binarization Approach for Fast Online Top-K Passage Matching. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 102--108.

[9]

Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, and Irwin King. 2023 b. WSFE: Wasserstein Sub-graph Feature Encoder for Effective User Segmentation in Collaborative Filtering. arXiv preprint arXiv:2305.04410 (2023).

[10]

Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable Feature Learning for Networks. In Proc. of KDD.

Digital Library

[11]

Michael Gutmann and Aapo Hyv"arinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS.

[12]

Hakim Hafidi, Mounir Ghogho, Philippe Ciblat, and Ananthram Swami. 2020. Graphcl: Contrastive self-supervised learning of graph representations. ArXiv preprint.

[13]

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In Proc. of NeurIPS.

[14]

Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. 2022. Vision GNN: An Image is Worth Graph of Nodes. ArXiv preprint.

[15]

Kaveh Hassani and Amir Hosein Khas Ahmadi. 2020. Contrastive Multi-View Representation Learning on Graphs. In Proc. of ICML.

[16]

Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. ArXiv preprint.

[17]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In Proc. of ICLR.

[18]

Bolian Li, Baoyu Jing, and Hanghang Tong. 2022. Graph Communal Contrastive Learning. In Proceedings of the ACM Web Conference 2022.

Digital Library

[19]

Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper Insights Into Graph Convolutional Networks for Semi-Supervised Learning. In Proc. of AAAI.

[20]

Shuai Lin, Chen Liu, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. 2022. Prototypical graph contrastive learning. IEEE Transactions on Neural Networks and Learning Systems.

[21]

Shuai Lin, Pan Zhou, Zi-Yuan Hu, Shuojia Wang, Ruihui Zhao, Yefeng Zheng, Liang Lin, Eric Xing, and Xiaodan Liang. 2021. Prototypical graph contrastive learning. ArXiv preprint.

[22]

Jan R Magnus. 1998. HANDBOOK OF MATRICES: H. Lütkepohl, John Wiley and Sons, 1996. Econometric Theory.

[23]

Costas Mavromatis and George Karypis. 2021. Graph InfoClust: Maximizing Coarse-Grain Mutual Information in Graphs. In Proc. of KDD.

Digital Library

[24]

Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-Based Recommendations on Styles and Substitutes. In Proc. of SIGIR.

[25]

Péter Mernyei and Cua tua lina Cangea. 2020. Wiki-cs: A wikipedia-based benchmark for graph neural networks. ArXiv preprint.

[26]

Diego P. P. Mesquita, Amauri H. Souza Jr., and Samuel Kaski. 2020. Rethinking pooling in graph neural networks. In Proc. of NeurIPS.

[27]

Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. 2017. graph2vec: Learning Distributed Representations of Graphs. ArXiv preprint.

[28]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kö pf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proc. of NeurIPS.

[29]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In Proc. of KDD.

Digital Library

[30]

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. In Proc. of KDD.

Digital Library

[31]

Marian-Andrei Rizoiu, Timothy Graham, Rui Zhang, Yifei Zhang, Robert Ackland, and Lexing Xie. 2018. # debatenight: The role and influence of socialbots on twitter during the 1st 2016 us presidential debate. In Proc. of AAAI.

[32]

Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. 2009. Efficient graphlet kernels for large graph comparison. In Proc. of AISTATS.

[33]

David I Shuman, Benjamin Ricaud, and Pierre Vandergheynst. 2012. A windowed graph Fourier transform. In 2012 IEEE Statistical Signal Processing Workshop (SSP).

[34]

Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In Proc. of WWW.

Digital Library

[35]

Zixing Song and Irwin King. 2022. Hierarchical Heterogeneous Graph Attention Network for Syntax-Aware Summarization. In Proc. of AAAI.

[36]

Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. 2021. Semi-supervised multi-label learning for graph-structured data. In Proc. of CIKM.

Digital Library

[37]

Zixing Song, Yifei Zhang, and Irwin King. 2022. Towards an optimal asymmetric graph structure for robust semi-supervised node classification. In Proc. of KDD.

Digital Library

[38]

Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. 2019. Deeply-Supervised Knowledge Synergy. In Proc. of CVPR.

[39]

Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. 2020. InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In Proc. of ICLR.

[40]

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019. Deep Graph Infomax. In Proc. of ICLR.

[41]

Menglin Yang, Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. 2022a. Hicf: Hyperbolic informative collaborative filtering. In Proc. of KDD.

Digital Library

[42]

Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. 2021. Discrete-time temporal network embedding via implicit hierarchical learning in hyperbolic space. In Proc. of KDD.

Digital Library

[43]

Menglin Yang, Min Zhou, Jiahong Liu, Defu Lian, and Irwin King. 2022b. HRCF: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM Web Conference 2022.

Digital Library

[44]

Menglin Yang, Min Zhou, Hui Xiong, and Irwin King. 2022c. Hyperbolic Temporal Network Embedding. IEEE Transactions on Knowledge and Data Engineering.

Digital Library

[45]

Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. 2018. Hierarchical Graph Representation Learning with Differentiable Pooling. In Proc. of NeurIPS.

[46]

Yifei Zhang, Dun Zeng, Jinglong Luo, Zenglin Xu, and Irwin King. 2023. A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacy. ArXiv preprint.

[47]

Yifei Zhang and Hao Zhu. 2019. Doc2hash: Learning Discrete Latent variables for Documents Retrieval. In Proc. of NAACL.

[48]

Yifei Zhang and Hao Zhu. 2020a. Additively homomorphical encryption based deep neural network for asymmetrically collaborative machine learning. arXiv preprint arXiv:2007.06849 (2020).

[49]

Yifei Zhang and Hao Zhu. 2020b. Discrete Wasserstein Autoencoders for Document Retrieval. In Proc. of ICASSP.

[50]

Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, and Irwin King. 2022a. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022.

Digital Library

[51]

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2022b. COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning. In Proc. of KDD.

Digital Library

[52]

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2022c. Spectral Feature Augmentation for Graph Contrastive Learning and Beyond. ArXiv preprint.

[53]

Gangming Zhao, Weifeng Ge, and Yizhou Yu. 2021. GraphFPN: Graph Feature Pyramid Network for Object Detection. In Proc. of ICCV.

[54]

Hao Zhu and Piotr Koniusz. 2021. Simple spectral graph convolution. In International conference on learning representations.

[55]

Hao Zhu and Piotr Koniusz. 2022a. EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9078--9088.

[56]

Hao Zhu and Piotr Koniusz. 2022b. Generalized laplacian eigenmaps. Advances in Neural Information Processing Systems, Vol. 35 (2022), 30783--30797.

[57]

Hao Zhu, Ke Sun, and Peter Koniusz. 2021a. Contrastive Laplacian Eigenmaps. In Proc. of NeurIPS.

[58]

Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. 2021b. An Empirical Study of Graph Contrastive Learning. ArXiv preprint.

[59]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. ArXiv preprint.

[60]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021c. Graph contrastive learning with adaptive augmentation. In Proc. of WWW. io

Digital Library

Cited By

Song ZYang XZhang YFu XXu ZKing ILarson K(2024)A systematic survey on federated semi-supervised learningProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/911(8244-8252)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/911
Lu YZhao WSun NWang JLarson K(2024)Enhancing multimodal knowledge graph representation learning through triple contrastive learningProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/659(5963-5971)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/659
Zhang YZhu HSong ZChen YFu XMeng ZKoniusz PKing IBaeza-Yates RBonchi F(2024)Geometric View of Soft Decorrelation in Self-Supervised LearningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671914(4338-4349)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671914
Show More Cited By

Index Terms

Contrastive Cross-scale Graph Knowledge Synergy
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Cross-view graph contrastive learning with hypergraph
Abstract
Graph contrastive learning (GCL) provides a new perspective to alleviate the reliance on labeled data for graph representation learning. Recent efforts on GCL leverage various graph augmentation strategies, i.e., node dropping and edge masking, ...
Highlights
- We proposed that hypergraphs are used as a paradigm to enhance graph contrastive learning.
- We propose a novel diffusion model-based fusion mechanism that aligns the positive examples.
- Our experimental results all exceed existing ...
Cross-scale contrastive triplet networks for graph representation learning
Abstract
Graph representation learning aims to learn low-dimensional representation for the graph, which has played a vital role in real-world applications. Without requiring additional labeled data, contrastive learning based graph representation ...
Highlights
- A contrastive learning framework for extracting multiple information is proposed.
- A combination of contextual and intrinsic information of graph data is used for node representation.
- The experimental results superior to the latest ...
Graph contrastive learning with min-max mutual information
Abstract
Graph contrastive learning has achieved rapid development in learning representations from graph-structured data, which aims to maximize the mutual information between two representations learned from different augmented views of a graph. However,...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2023

5996 pages

ISBN:9798400701030

DOI:10.1145/3580305

General Chairs:
Ambuj Singh
UC Santa Barbara, USA
,
Yizhou Sun
UC Los Angeles, USA
,
Program Chairs:
Leman Akoglu
Carnegie Mellon University, USA
,
Dimitrios Gunopulos
University of Athens, Greece
,
Xifeng Yan
UC Santa Barbara, USA
,
Ravi Kumar
Google, USA
,
Fatma Ozcan
Google, USA
,
Jieping Ye
Alibaba DAMO Academy

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CUHK 2410021, Research Impact Fund
National Key Research and Development Program of China

Conference

KDD '23

Sponsor:

KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 6 - 10, 2023

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
676
Total Downloads

Downloads (Last 12 months)282
Downloads (Last 6 weeks)21

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song ZYang XZhang YFu XXu ZKing ILarson K(2024)A systematic survey on federated semi-supervised learningProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/911(8244-8252)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/911
Lu YZhao WSun NWang JLarson K(2024)Enhancing multimodal knowledge graph representation learning through triple contrastive learningProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/659(5963-5971)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/659
Zhang YZhu HSong ZChen YFu XMeng ZKoniusz PKing IBaeza-Yates RBonchi F(2024)Geometric View of Soft Decorrelation in Self-Supervised LearningProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671914(4338-4349)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671914
Liu YZhang HHe TZheng TZhao J(2024)Bootstrap Latents of Nodes and Neighbors for Graph Self-supervised LearningMachine Learning and Knowledge Discovery in Databases. Research Track10.1007/978-3-031-70352-2_5(76-92)Online publication date: 22-Aug-2024
https://doi.org/10.1007/978-3-031-70352-2_5
Song ZZhang YKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Optimal block-wise asymmetric graph construction for graph-based semi-supervised learningProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669237(71135-71149)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669237
Zhang YZhu HChen YSong ZKoniusz PKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)Mitigating the popularity bias of graph collaborative filteringProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669074(67533-67550)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669074
Song ZZhang YKing IOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)No change, no gainProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668180(47511-47526)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3668180
Song ZZhang YKing IFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Towards Fair Financial Services for All: A Temporal GNN Approach for Individual Fairness on Transaction NetworksProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615091(2331-2341)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615091
Wu YChen YYin ZDing WKing I(2023)A survey on graph embedding techniques for biomedical dataInformation Fusion10.1016/j.inffus.2023.101909100:COnline publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.inffus.2023.101909

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten