More Web Proxy on the site http://driver.im/

research-article

Self-Supervised Teaching and Learning of Representations on Graphs

Authors:

Feng XiaAuthors Info & Claims

WWW '23: Proceedings of the ACM Web Conference 2023

Pages 489 - 498

https://doi.org/10.1145/3543507.3583441

Published: 30 April 2023 Publication History

Abstract

Recent years have witnessed significant advances in graph contrastive learning (GCL), while most GCL models use graph neural networks as encoders based on supervised learning. In this work, we propose a novel graph learning model called GraphTL, which explores self-supervised teaching and learning of representations on graphs. One critical objective of GCL is to retain original graph information. For this purpose, we design an encoder based on the idea of unsupervised dimensionality reduction of locally linear embedding (LLE). Specifically, we map one iteration of the LLE to one layer of the network. To guide the encoder to better retain the original graph information, we propose an unbalanced contrastive model consisting of two views, which are the learning view and the teaching view, respectively. Furthermore, we consider the nodes that are identical in muti-views as positive node pairs, and design the node similarity scorer so that the model can select positive samples of a target node. Extensive experiments have been conducted over multiple datasets to evaluate the performance of GraphTL in comparison with baseline models. Results demonstrate that GraphTL can reduce distances between similar nodes while preserving network topological and feature information, yielding better performance in node classification.

References

[1]

Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V Chawla. 2021. Graph Barlow Twins: A self-supervised representation learning framework for graphs. arXiv preprint arXiv:2106.02466(2021).

[2]

Jie Chen, Tengfei Ma, and Cao Xiao. 2018. FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling. In International Conference on Learning Representations.

[3]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. PMLR, 1597–1607.

[4]

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289(2015).

[5]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in Neural Information Processing Systems 29 (2016).

[6]

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In International Conference on Machine Learning. PMLR, 1263–1272.

[7]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 249–256.

[8]

Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 855–864.

Digital Library

[9]

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems 30 (2017).

[10]

Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning. PMLR, 4116–4126.

[11]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision. 1026–1034.

Digital Library

[12]

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations.

[13]

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems 33 (2020), 22118–22133.

[14]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. PMLR, 448–456.

[15]

Amir Hosein Khasahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. 2019. Memory-Based Graph Networks. In International Conference on Learning Representations.

[16]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).

[17]

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907(2016).

[18]

Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. 2019. Graph matching networks for learning the similarity of graph structured objects. In International Conference on Machine Learning. PMLR, 3835–3845.

[19]

Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering (2021).

[20]

Ziyang Liu, Hao Feng, and Chaokun Wang. 2021. Rethinking Temperature in Graph Contrastive Learning. (2021).

[21]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).

[22]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26 (2013).

[23]

Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4004–4012.

[24]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748(2018).

[25]

Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. 2020. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020. 259–270.

Digital Library

[26]

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 701–710.

Digital Library

[27]

Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020. Gcc: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1150–1160.

Digital Library

[28]

Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. 2018. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 459–467.

Digital Library

[29]

Jing Ren, Feng Xia, Ivan Lee, Azadeh Noori Hoshyar, and Charu C. Aggarwal. 2022. Graph Learning for Anomaly Analytics: Algorithms, Applications, and Challenges. ACM Transactions on Intelligent Systems and Technology (2022). https://doi.org/10.1145/3570906

Digital Library

[30]

Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. 2017. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 385–394.

Digital Library

[31]

Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2019. DropEdge: Towards Deep Graph Convolutional Networks on Node Classification. In International Conference on Learning Representations.

[32]

Sam T Roweis and Lawrence K Saul. 2000. Nonlinear dimensionality reduction by locally linear embedding. science 290, 5500 (2000), 2323–2326.

[33]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition. 815–823.

[34]

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI Magazine 29, 3 (2008), 93–93.

Digital Library

[35]

Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems 29 (2016).

[36]

Fan-Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. 2019. InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In International Conference on Learning Representations.

[37]

Ke Sun, Lei Wang, Bo Xu, Wenhong Zhao, Shyh Wei Teng, and Feng Xia. 2020. Network Representation Learning: From Traditional Feature Learning to Deep Learning. IEEE Access 8(2020), 205600–205617. https://doi.org/10.1109/ACCESS.2020.3037118

[38]

Ke Sun, Feng Xia, Jiaying Liu, Bo Xu, Vidya Saikrishna, and Charu C. Aggarwal. 2022. Attributed Graph Force Learning. IEEE Transactions on Neural Networks and Learning Systems (2022). https://doi.org/10.1109/TNNLS.2022.3221100

[39]

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web. 1067–1077.

Digital Library

[40]

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. 2021. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop on Geometrical and Topological Representation Learning.

[41]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of Machine Learning Research 9, 11 (2008).

[42]

Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In International Conference on Learning Representations.

[43]

Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2018. Deep graph infomax. arXiv preprint arXiv:1809.10341(2018).

[44]

Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. 2019. Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks. arXiv preprint arXiv:1909.01315(2019).

[45]

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019. Simplifying graph convolutional networks. In International Conference on Machine Learning. PMLR, 6861–6871.

[46]

Lirong Wu, Haitao Lin, Cheng Tan, Zhangyang Gao, and Stan Z Li. 2021. Self-supervised learning on graphs: Contrastive, generative, or predictive. IEEE Transactions on Knowledge and Data Engineering (2021).

[47]

Feng Xia, Jiaying Liu, Hansong Nie, Yonghao Fu, Liangtian Wan, and Xiangjie Kong. 2020. Random Walks: A Review of Algorithms and Applications. IEEE Transactions on Emerging Topics in Computational Intelligence 4, 2(2020), 95–107. https://doi.org/10.1109/TETCI.2019.2952908

[48]

Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz, Liangtian Wan, Shirui Pan, and Huan Liu. 2021. Graph Learning: A Survey. IEEE Transactions on Artificial Intelligence 2, 2(2021), 109–127. https://doi.org/10.1109/TAI.2021.3076021

[49]

Feng Xia, Lei Wang, Tao Tang, Xin Chen, Xiangjie Kong, Giles Oatley, and Irwin King. 2022. CenGCN: Centralized Convolutional Networks with Vertex Imbalance for Scale-Free Graphs. IEEE Transactions on Knowledge and Data Engineering (2022). https://doi.org/10.1109/TKDE.2022.3149888

Digital Library

[50]

Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. 2022. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).

[51]

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How Powerful are Graph Neural Networks¿. In International Conference on Learning Representations.

[52]

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems 33 (2020), 5812–5823.

[53]

Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. 2020. L2-gcn: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2127–2135.

[54]

Hanlin Zhang, Shuai Lin, Weiyang Liu, Pan Zhou, Jian Tang, Xiaodan Liang, and Eric P Xing. 2020. Iterative Graph Self-Distillation. arXiv e-prints (2020), arXiv–2010.

[55]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131(2020).

[56]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2021. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021. 2069–2080.

Digital Library

[57]

Marinka Zitnik and Jure Leskovec. 2017. Predicting multicellular function through multi-layer tissue networks. Bioinformatics 33, 14 (2017), i190–i198.

Cited By

Sun LLi CRen YZhang Y(2024)A Multitask Dynamic Graph Attention Autoencoder for Imbalanced Multilabel Time Series ClassificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2024.336906435:9(11829-11842)Online publication date: Sep-2024
https://doi.org/10.1109/TNNLS.2024.3369064
Niu CPang GChen L(2024)Affinity Uncertainty-Based Hard Negative Mining in Graph Contrastive LearningIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.333977035:9(11681-11691)Online publication date: Sep-2024
https://doi.org/10.1109/TNNLS.2023.3339770
Zhang PBai HDai YLing HZhang K(2024)Dynamic Denoising of Contrastive Learning for GNN-based Node Embedding2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650398(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650398

Index Terms

Self-Supervised Teaching and Learning of Representations on Graphs
1. Computing methodologies
  1. Artificial intelligence
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
    2. Machine learning approaches
      1. Neural networks

Recommendations

Graph Self-supervised Learning with Augmentation-aware Contrastive Learning
WWW '23: Proceedings of the ACM Web Conference 2023

Graph self-supervised learning aims to mine useful information from unlabeled graph data, and has been successfully applied to pre-train graph representations. Many existing approaches use contrastive learning to learn powerful embeddings by learning ...
A noise-resistant graph neural network by semi-supervised contrastive learning
Abstract
Graph neural networks (GNNs) have been widely applied for representation learning on the graph data in real applications, but few of them are designed to conduct representation learning on the graph data with noisy labels. Its key challenge is ...
Highlights
- Propose a new noise-resistant graph neural network to conduct representation learning on the graph data with noisy labels.
- Propose a semi-supervised contrastive learning constraint to push noisy nodes away from unlabeled nodes in ...
Self-supervised reconstructed graph learning for link prediction in bipartite graphs
Abstract
Graph Neural Network(GNN) has achieved remarkable performance in classification tasks due to its strong distinctive power of different graph topologies. However, traditional GNNs face great limitations in link prediction tasks since they learn ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '23: Proceedings of the ACM Web Conference 2023

April 2023

4293 pages

ISBN:9781450394161

DOI:10.1145/3543507

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
481
Total Downloads

Downloads (Last 12 months)129
Downloads (Last 6 weeks)6

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun LLi CRen YZhang Y(2024)A Multitask Dynamic Graph Attention Autoencoder for Imbalanced Multilabel Time Series ClassificationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2024.336906435:9(11829-11842)Online publication date: Sep-2024
https://doi.org/10.1109/TNNLS.2024.3369064
Niu CPang GChen L(2024)Affinity Uncertainty-Based Hard Negative Mining in Graph Contrastive LearningIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.333977035:9(11681-11691)Online publication date: Sep-2024
https://doi.org/10.1109/TNNLS.2023.3339770
Zhang PBai HDai YLing HZhang K(2024)Dynamic Denoising of Contrastive Learning for GNN-based Node Embedding2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650398(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650398
Zhang LLiang SWan L(2024)A multi-view graph contrastive learning framework for deciphering spatially resolved transcriptomics dataBriefings in Bioinformatics10.1093/bib/bbae25525:4Online publication date: 27-May-2024
https://doi.org/10.1093/bib/bbae255

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents