More Web Proxy on the site http://driver.im/

research-article

Open access

Geometric View of Soft Decorrelation in Self-Supervised Learning

Authors:

Irwin KingAuthors Info & Claims

KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 4338 - 4349

https://doi.org/10.1145/3637528.3671914

Published: 24 August 2024 Publication History

Abstract

Contrastive learning, a form of Self-Supervised Learning (SSL), typically consists of an alignment term and a regularization term. The alignment term minimizes the distance between the embeddings of a positive pair, while the regularization term prevents trivial solutions and expresses prior beliefs about the embeddings. As a widely used regularization technique, soft decorrelation has been employed by several non-contrastive SSL methods to avoid trivial solutions. While the decorrelation term is designed to address the issue of dimensional collapse, we find that it fails to achieve this goal theoretically and experimentally. Based on such a finding, we extend the soft decorrelation regularization to minimize the distance between the covariance matrix and an identity matrix. We provide a new perspective on the geometric distance between positive definite matrices to investigate why the soft decorrelation cannot efficiently solve the dimensional collapse. Furthermore, we construct a family of loss functions utilizing the Bregman Matrix Divergence (BMD), with the soft decorrelation representing a specific instance within this family. We prove that a loss function (LogDet) in this family can solve the issue of dimensional collapse. Our novel loss functions based on BMD exhibit superior performance compared to the soft decorrelation and other baseline techniques, as demonstrated by experimental results on graph and image datasets.

References

[1]

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. 2019. Implicit Regularization in Deep Matrix Factorization. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 7411--7422. https://proceedings.neurips.cc/paper/2019/hash/c0c783b5fc0d7d808f1d14a6e9c8280d-Abstract.html

[2]

Philip Bachman, R. Devon Hjelm, and William Buchwalter. 2019. Learning Representations by Maximizing Mutual Information Across Views. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8--14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 15509--15519. https://proceedings.neurips.cc/paper/2019/hash/ddf354219aac374f1d40b7e760ee5bb7-Abstract.html

[3]

Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25--29, 2022. OpenReview.net. https://openreview.net/forum?id=xm6YD62D1Ub

[4]

Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V Chawla. 2022. Graph Barlow Twins: A self-supervised representation learning framework for graphs. Knowledge-Based Systems, Vol. 256 (2022), 109631.

Digital Library

[5]

Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=r1ZdKJ-0W

[6]

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/70feb62b69f16e0238f741fab228fec2-Abstract.html

[7]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13--18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 1597--1607. http://proceedings.mlr.press/v119/chen20j.html

[8]

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. Big Self-Supervised Models are Strong Semi-Supervised Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/fcbc95ccdd551da181207c0c1400c655-Abstract.html

[9]

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. ArXiv preprint, Vol. abs/2003.04297 (2020). https://arxiv.org/abs/2003.04297

[10]

Xinlei Chen and Kaiming He. 2021. Exploring Simple Siamese Representation Learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19--25, 2021. Computer Vision Foundation / IEEE, 15750--15758. https://doi.org/10.1109/CVPR46437.2021.01549

[11]

Yankai Chen, Yixiang Fang, Qiongyan Wang, Xin Cao, and Irwin King. 2024. Deep Structural Knowledge Exploitation and Synergy for Estimating Node Importance Value on Heterogeneous Information Networks. In AAAI. 8302--8310.

[12]

Yankai Chen, Yixiang Fang, Yifei Zhang, and Irwin King. 2023. Bipartite graph convolutional hashing for effective and efficient top-n search in hamming space. In Proceedings of the ACM Web Conference 2023. 3164--3172.

Digital Library

[13]

Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King. 2022. Learning binarized graph representations with multi-faceted quantization reinforcement for top-k recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 168--178.

Digital Library

[14]

Yankai Chen, Yifei Zhang, Huifeng Guo, Ruiming Tang, and Irwin King. 2022. An Effective Post-training Embedding Binarization Approach for Fast Online Top-K Passage Matching. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online only, 102--108. https://aclanthology.org/2022.aacl-short.14

[15]

Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, and Irwin King. 2023. WSFE: wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2521--2525.

Digital Library

[16]

Anoop Cherian, Suvrit Sra, Arindam Banerjee, and Nikolaos Papanikolopoulos. 2012. Jensen-bregman logdet divergence with application to efficient similarity search for covariance matrices. IEEE transactions on pattern analysis and machine intelligence, Vol. 35, 9 (2012), 2161--2174.

[17]

Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. 2022. solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. J. Mach. Learn. Res., Vol. 23 (2022), 56:1--56:6. http://jmlr.org/papers/v23/21--1155.html

[18]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20--25 June 2009, Miami, Florida, USA. IEEE Computer Society, 248--255. https://doi.org/10.1109/CVPR.2009.5206848

[19]

Carl Doersch, Abhinav Gupta, and Alexei A. Efros. 2015. Unsupervised Visual Representation Learning by Context Prediction. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7--13, 2015. IEEE Computer Society, 1422--1430. https://doi.org/10.1109/ICCV.2015.167

Digital Library

[20]

Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. 2021. Whitening for Self-Supervised Representation Learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18--24 July 2021, Virtual Event (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 3015--3024. http://proceedings.mlr.press/v139/ermolov21a.html

[21]

Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with PyTorch Geometric. ArXiv preprint, Vol. abs/1903.02428 (2019). https://arxiv.org/abs/1903.02428

[22]

Spyros Gidaris, Praveer Singh, and Nikos Komodakis. 2018. Unsupervised Representation Learning by Predicting Image Rotations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=S1v4N2l0-

[23]

Gene H Golub and Charles F Van Loan. 2013. Matrix computations. JHU press.

[24]

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual, Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/f3ada80d5c4ee70142b17b8192b2958e-Abstract.html

[25]

Maryam Haghighat, Peyman Moghadam, Shaheer Mohamed, and Piotr Koniusz. 2024. Pre-training with Random Orthogonal Projection Image Modeling. In International Conference on Learning Representations (ICLR).

[26]

Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley. 2017. Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods. IEEE transactions on pattern analysis and machine intelligence, Vol. 40, 1 (2017), 48--62.

[27]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13--19, 2020. IEEE, 9726--9735. https://doi.org/10.1109/CVPR42600.2020.00975

[28]

Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. 2021. On Feature Decorrelation in Self-Supervised Learning. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10--17, 2021. IEEE, 9578--9588. https://doi.org/10.1109/ICCV48922.2021.00946

[29]

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. 2022. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25--29, 2022. OpenReview.net. https://openreview.net/forum?id=YevsQ05DEN7

[30]

Piotr Koniusz, Fei Yan, Philippe-Henri Gosselin, and Krystian Mikolajczyk. 2013. Higher-order occurrence pooling on mid-and low-level features: Visual concept detection. Tech. Report (2013).

[31]

Piotr Koniusz and Hongguang Zhang. 2022. Power Normalizations in Fine-Grained Image, Few-Shot Image and Graph Classification. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 44, 2 (2022), 591--609.

Digital Library

[32]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).

[33]

Brian Kulis, Mátyás A Sustik, and Inderjit S Dhillon. 2009. Low-Rank Kernel Learning with Bregman Matrix Divergences. Journal of Machine Learning Research, Vol. 10, 2 (2009).

[34]

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14. Springer, 577--593.

[35]

Kanglin Liu, Guoping Qiu, Wenming Tang, and Fei Zhou. 2019. Spectral Regularization for Combating Mode Collapse in GANs. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 6381--6389. https://doi.org/10.1109/ICCV.2019.00648

[36]

Yueen Ma, Zixing Song, Xuming Hu, Jingjing Li, Yifei Zhang, and Irwin King. 2023. Graph component contrastive learning for concept relatedness estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 13362--13370.

Digital Library

[37]

Julian J. McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-Based Recommendations on Styles and Substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9--13, 2015, Ricardo Baeza-Yates, Mounia Lalmas, Alistair Moffat, and Berthier A. Ribeiro-Neto (Eds.). ACM, 43--52. https://doi.org/10.1145/2766462.2767755

Digital Library

[38]

Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part VI. Springer, 69--84.

[39]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. ArXiv preprint, Vol. abs/1807.03748 (2018). https://arxiv.org/abs/1807.03748

[40]

Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27--30, 2016. IEEE Computer Society, 2536--2544. https://doi.org/10.1109/CVPR.2016.278

[41]

Arian Prabowo, Hao Xue, Wei Shao, Piotr Koniusz, and Flora D. Salim. 2023. Traffic forecasting on new roads using spatial contrastive pre-training (SCPT). Data Min. Knowl. Discov., Vol. 38, 3 (sep 2023), 913--937. https://doi.org/10.1007/s10618-023-00982-0

Digital Library

[42]

Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference. IEEE, 606--610.

[43]

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. 2018. Pitfalls of graph neural network evaluation. ArXiv preprint, Vol. abs/1811.05868 (2018). https://arxiv.org/abs/1811.05868

[44]

Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. 2021. Semi-supervised multi-label learning for graph-structured data. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 1723--1733.

Digital Library

[45]

Zixing Song, Yifei Zhang, and Irwin King. 2022. Towards an optimal asymmetric graph structure for robust semi-supervised node classification. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 1656--1665.

Digital Library

[46]

Zixing Song, Yifei Zhang, and Irwin King. 2024. No Change, No Gain: Empowering Graph Neural Networks with Expected Model Change Maximization for Active Learning. Advances in Neural Information Processing Systems, Vol. 36 (2024).

[47]

Zixing Song, Yifei Zhang, and Irwin King. 2024. Optimal Block-wise Asymmetric Graph Construction for Graph-based Semi-supervised Learning. Advances in Neural Information Processing Systems, Vol. 36 (2024).

[48]

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019. Deep Graph Infomax. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6--9, 2019. OpenReview.net. https://openreview.net/forum?id=rklz9iAcKQ

[49]

Feng Wang and Huaping Liu. 2021. Understanding the Behaviour of Contrastive Loss. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19--25, 2021. Computer Vision Foundation / IEEE, 2495--2504. https://doi.org/10.1109/CVPR46437.2021.00252

[50]

Tongzhou Wang and Phillip Isola. 2020. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13--18 July 2020, Virtual Event (Proceedings of Machine Learning Research, Vol. 119). PMLR, 9929--9939. http://proceedings.mlr.press/v119/wang20k.html

[51]

Yaozu Wu, Yankai Chen, Zhishuai Yin, Weiping Ding, and Irwin King. 2023. A survey on graph embedding techniques for biomedical data: Methods and applications. Information Fusion, Vol. 100 (2023), 101909.

Digital Library

[52]

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9653--9663.

[53]

Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2016. Revisiting Semi-Supervised Learning with Graph Embeddings. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19--24, 2016 (JMLR Workshop and Conference Proceedings, Vol. 48), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). JMLR.org, 40--48. http://proceedings.mlr.press/v48/yanga16.html

[54]

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18--24 July 2021, Virtual Event (Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 12310--12320. http://proceedings.mlr.press/v139/zbontar21a.html

[55]

Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S. Yu. 2021. From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6--14, 2021, virtual, Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 76--89. https://proceedings.neurips.cc/paper/2021/hash/00ac8ed3b4327bdd4ebbebcb2ba10a00-Abstract.html

[56]

Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. 2022. Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25--29, 2022. OpenReview.net. https://openreview.net/forum?id=RAW9tCdVxLj

[57]

Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. 2022 d. Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25--29, 2022. OpenReview.net. https://openreview.net/forum?id=RAW9tCdVxLj

[58]

Xinni Zhang, Yankai Chen, Chenhao Ma, Yixiang Fang, and Irwin King. 2024. Influential Exemplar Replay for Incremental Learning in Recommender Systems. In AAAI, Vol. 38. 9368--9376.

Digital Library

[59]

Yifei Zhang, Yankai Chen, Zixing Song, and Irwin King. 2023. Contrastive cross-scale graph knowledge synergy. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3422--3433.

Digital Library

[60]

Yifei Zhang and Hao Zhu. 2019. Doc2hash: Learning Discrete Latent variables for Documents Retrieval. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 2235--2240. https://doi.org/10.18653/v1/N19--1232

[61]

Yifei Zhang and Hao Zhu. 2020. Discrete Wasserstein Autoencoders for Document Retrieval. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4--8, 2020. IEEE, 8159--8163. https://doi.org/10.1109/ICASSP40776.2020.9053129

[62]

Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, and Irwin King. 2022. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022. 1331--1339.

Digital Library

[63]

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2022. COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning. In KDD.

Digital Library

[64]

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. 2023. Spectral feature augmentation for graph contrastive learning and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 11289--11297.

Digital Library

[65]

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, Irwin King, et al. 2023. Mitigating the popularity bias of graph collaborative filtering: A dimensional collapse perspective. Advances in Neural Information Processing Systems, Vol. 36 (2023), 67533--67550.

[66]

Hao Zhu and Piotr Koniusz. 2021. REFINE: Random RangE FInder for Network Embedding (CIKM '21). Association for Computing Machinery, New York, NY, USA, 3682--3686. https://doi.org/10.1145/3459637.3482168

Digital Library

[67]

Hao Zhu and Piotr Koniusz. 2022. Generalized Laplacian Eigenmaps. In Advances in Neural Information Processing Systems, Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.). https://openreview.net/forum?id=HjicdpP-Nth

[68]

Hao Zhu, Ke Sun, and Piotr Koniusz. 2021. Contrastive Laplacian Eigenmaps. In Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (Eds.). https://openreview.net/forum?id=iLn-bhP-kKH

[69]

Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020. Deep graph contrastive representation learning. ArXiv preprint, Vol. abs/2006.04131 (2020). https://arxiv.org/abs/2006.04131

Cited By

Li YMoghadam PPeng CYe NKoniusz PNejdl WAuer SKarras OCha MMoens MNajork M(2025)Inductive Graph Few-shot Class Incremental LearningProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703578(466-474)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703578
Dong JKoniusz PChen JOng Y(2024)Adversarially Robust Distillation by Reducing the Student-Teacher Variance GapComputer Vision – ECCV 202410.1007/978-3-031-73235-5_6(92-111)Online publication date: 30-Sep-2024
https://doi.org/10.1007/978-3-031-73235-5_6

Index Terms

Geometric View of Soft Decorrelation in Self-Supervised Learning

Index terms have been assigned to the content through auto-classification.

Recommendations

Self-supervised semi-supervised nonnegative matrix factorization for data clustering
Highlights
- A self-supervised semi-supervised NMF method is proposed for data clustering.
- ...
Abstract
Semi-supervised nonnegative matrix factorization exploits the strengths of matrix factorization in successfully learning part-based representation and is also able to achieve high learning performance when facing a scarcity of labeled ...
Generalized self-supervised contrastive learning with bregman divergence for image recognition
Highlights
- A generalized contrastive learning framework is proposed.
- Bregman divergence ...
Abstract
Contrastive learning techniques continue to receive a lot of attention in the self-supervised learning area. Specifically, the learned distance features can be further utilized to capture the distance between latent features in the ...
Improving Few-Shot Image Classification with Self-supervised Learning
Cloud Computing – CLOUD 2022
Abstract
Few-Shot Image Classification (FSIC) aims to learn an image classifier with only a few training samples. The key challenge of few-shot image classification is to learn this classifier with scarce labeled data. To tackle the issue, we leverage the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2024

6901 pages

ISBN:9798400704901

DOI:10.1145/3637528

General Chairs:
Ricardo Baeza-Yates
Northeastern University, USA
,
Francesco Bonchi
CENTAI / Eurecat, Italy

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Research Grants Council of the Hong Kong Special Administrative Region, China

Conference

KDD '24

Sponsor:

KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 25 - 29, 2024

Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
678
Total Downloads

Downloads (Last 12 months)678
Downloads (Last 6 weeks)85

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li YMoghadam PPeng CYe NKoniusz PNejdl WAuer SKarras OCha MMoens MNajork M(2025)Inductive Graph Few-shot Class Incremental LearningProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining10.1145/3701551.3703578(466-474)Online publication date: 10-Mar-2025
https://dl.acm.org/doi/10.1145/3701551.3703578
Dong JKoniusz PChen JOng Y(2024)Adversarially Robust Distillation by Reducing the Student-Teacher Variance GapComputer Vision – ECCV 202410.1007/978-3-031-73235-5_6(92-111)Online publication date: 30-Sep-2024
https://doi.org/10.1007/978-3-031-73235-5_6

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten