Just Accepted
Characterizing and Understanding HGNN Training on GPUs
Accepted on 23 October 2024
Abstract
Owing to their remarkable representation capabilities for heterogeneous graph data, Heterogeneous Graph Neural Networks (HGNNs) have been widely adopted in many critical real-world domains such as recommendation systems and medical analysis. Prior to their practical application, identifying the optimal HGNN model parameters tailored to specific tasks through extensive training is a time-consuming and costly process. To enhance the efficiency of HGNN training, it is essential to characterize and analyze the execution semantics and patterns within the training process to identify performance bottlenecks. In this study, we conduct a comprehensive quantification and in-depth analysis of two mainstream HGNN training scenarios, including single-GPU and multi-GPU distributed training. Based on the characterization results, we reveal the performance bottlenecks and their underlying causes in different HGNN training scenarios and propose optimization guidelines from both software and hardware perspectives.
References
[1]
Basmah Altaf, Uchenna Akujuobi, Lu Yu, and Xiangliang Zhang. 2019. Dataset Recommendation via Variational Graph Autoencoder. In 2019 IEEE International Conference on Data Mining (ICDM). 11–20.
[2]
Rui Bing, Guan Yuan, Mu Zhu, Fanrong Meng, Huifang Ma, and Shaojie Qiao. 2023. Heterogeneous graph neural networks analysis: a survey of techniques, evaluations and applications. Artificial Intelligence Review 56, 8 (2023), 8003–8042.
[3]
Dan Chen, Haiheng He, Hai Jin, et al. 2023. MetaNMP: Leveraging Cartesian-Like Product to Accelerate HGNNs with Near-Memory Processing. In Proceedings of the 50th Annual International Symposium on Computer Architecture (Orlando, FL, USA) (ISCA ’23). Association for Computing Machinery, New York, NY, USA, Article 56, 13 pages.
[4]
Shaohua Fan, Chuan Shi, and Xiao Wang. 2018. Abnormal Event Detection via Heterogeneous Information Network Embedding. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM ’18). Association for Computing Machinery, New York, NY, USA, 1483–1486.
[5]
Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds.
[6]
Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King. 2020. Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding. In Proceedings of The Web Conference 2020. 2331–2341.
[7]
Zhangxiaowen Gong, Houxiang Ji, Yao Yao, Christopher W. Fletcher, Christopher J. Hughes, and Josep Torrellas. 2022. Graphite: optimizing graph neural networks on CPUs through cooperative software-hardware techniques. In Proceedings of the 49th Annual International Symposium on Computer Architecture (New York, New York) (ISCA ’22). Association for Computing Machinery, New York, NY, USA, 916–931.
[8]
T. J. Ham, L. Wu, N. Sundaram, N. Satish, and M. Martonosi. 2016. Graphicionado: A High-Performance and Energy-Efficient Accelerator for Graph Analytics. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 1–13.
[9]
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems 30 (2017).
[10]
Dengke Han, Meng Wu, Runzhen Xue, Mingyu Yan, Xiaochun Ye, and Dongrui Fan. 2024. ADE-HGNN: Accelerating HGNNs through Attention Disparity Exploitation. In Euro-Par 2024: Parallel Processing - 30th International Conference on Parallel and Distributed Computing, Madrid, Spain, August 25 - August 30, 2024, Proceedings (Lecture Notes in Computer Science). 91–106.
[11]
Shifu Hou, Yanfang Ye, Yangqiu Song, and Melih Abdulhayoglu. 2017. HinDroid: An Intelligent Android Malware Detection System Based on Structured Heterogeneous Information Network. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Halifax, NS, Canada) (KDD ’17). Association for Computing Machinery, New York, NY, USA, 1507–1515. https://doi.org/10.1145/3097983.3098026
[12]
Xin Huang, Jongryool Kim, Bradley Rees, and Chul-Ho Lee. 2022. Characterizing the Efficiency of Graph Neural Network Frameworks with a Magnifying Glass. In 2022 IEEE International Symposium on Workload Characterization (IISWC). 160–170.
[13]
Canghong Jin, Tao Ruan, Dexing Wu, Lei Xu, Tengran Dong, Tianyi Chen, Shuoping Wang, Yi Du, and Minghui Wu. 2021. HetGAT: a heterogeneous graph attention network for freeway traffic speed prediction. Journal of Ambient Intelligence and Humanized Computing (01 2021). https://doi.org/10.1007/s12652-020-02807-0
[14]
Sein Kim, Namkyeong Lee, Junseok Lee, Dongmin Hyun, and Chanyoung Park. 2023. Heterogeneous Graph Learning for Multi-Modal Medical Data Analysis. Proceedings of the AAAI Conference on Artificial Intelligence 37, 4(Jun. 2023), 5141–5150.
[15]
Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations, ICLR 2017.
[16]
Haiyang Lin, Mingyu Yan, Xiaocheng Yang, et al. 2022. Characterizing and Understanding Distributed GNN Training on GPUs. IEEE Computer Architecture Letters 21, 1 (2022), 21–24.
[17]
Haiyang Lin, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Shirui Pan, Wenguang Chen, and Yuan Xie. 2023. A Comprehensive Survey on Distributed Training of Graph Neural Networks. Proc. IEEE 111, 12 (2023), 1572–1606.
[18]
Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu. 2020. PaGraph: Scaling GNN training on large graphs via computation-aware caching. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC ’20). Association for Computing Machinery, New York, NY, USA, 401–415.
[19]
Xin Liu, Mingyu Yan, Lei Deng, Guoqi Li, Xiaochun Ye, and Dongrui Fan. 2022. Sampling Methods for Efficient Training of Graph Convolutional Networks: A Survey. IEEE/CAA Journal of Automatica Sinica 9, 2 (2022), 205–234.
[20]
Feng Luo, Yue Zhang, and Xiaoli Wang. 2021. IMAS++ An Intelligent Medical Analysis System Enhanced with Deep Graph Neural Networks. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 4754–4758.
[21]
Qingsong Lv, Ming Ding, Qiang Liu, et al. 2021. Are we really making much progress? Revisiting, benchmarking and refining heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1150–1160.
[22]
Zhengyang Lv, Mingyu Yan, Xin Liu, Mengyao Dong, Xiaochun Ye, Dongrui Fan, and Ninghui Sun. 2023. A Survey of Graph Pre-processing Methods: From Algorithmic to Hardware Perspectives. arxiv:2309.07581 [cs.AR]
[23]
Mahmoud Nazzal, Abdallah Khreishah, Joyoung Lee, Shaahin Angizi, Ala Al-Fuqaha, and Mohsen Guizani. 2024. Semi-decentralized Inference in Heterogeneous Graph Neural Networks for Traffic Demand Forecasting: An Edge-Computing Approach. IEEE Transactions on Vehicular Technology(2024), 1–16.
[24]
Zheng Qu, Dimin Niu, Shuangchen Li, Hongzhong Zheng, and Yuan Xie. 2023. TT-GNN: Efficient On-Chip Graph Neural Network Training via Embedding Reformation and Hardware Optimization. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (Toronto, ON, Canada) (MICRO ’23). Association for Computing Machinery, New York, NY, USA, 452–464.
[25]
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference. Springer, 593–607.
[26]
Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. 2016. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering 29, 1(2016), 17–37.
[27]
John Thorpe, Yifan Qiao, Jon Eyolfson, Shen Teng, Guanzhou Hu, Zhihao Jia, Jinliang Wei, Keval Vora, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu. 2021. Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads. In USENIX Symposium on Operating Systems Design and Implementation.
[28]
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations, ICLR 2018 (2018).
[29]
Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational Graph Attention Network for Aspect-based Sentiment Analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3229–3238.
[30]
Minjie Wang, Lingfan Yu, Zheng Da, Gan Quan, Gai Yu, Ye Zihao, et al. 2019. Deep graph library: Towards efficient and scalable deep learning on graphs. In ICLR.
[31]
Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and Philip S Yu. 2020. A survey on heterogeneous graph embedding: methods, techniques, applications and sources. arXiv preprint arXiv:2011.14867(2020).
[32]
Xiao Wang, Houye Ji, Chuan Shi, et al. 2019. Heterogeneous graph attention network. In The world wide web conference. 2022–2032.
[33]
Zhaokang Wang, Yunpan Wang, Chunfeng Yuan, Rong Gu, and Yihua Huang. 2021. Empirical analysis of performance bottlenecks in graph neural network training and inference with GPUs. Neurocomputing 446(2021), 165–191. https://doi.org/10.1016/j.neucom.2021.03.015
[34]
Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52, 4 (apr 2009), 65–76. https://doi.org/10.1145/1498765.1498785
[35]
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2020. Graph neural networks in recommender systems: a survey. ACM Computing Surveys (CSUR)(2020).
[36]
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. 2020. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems 32, 1(2020), 4–24.
[37]
Ru xia Liang, Qian Zhang, Jianqiang Wang, and Jie Lu. 2022. A Hierarchical Attention Network for Cross-Domain Group Recommendation. IEEE Transactions on Neural Networks and Learning Systems 35 (2022), 3859–3873.
[38]
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018. How powerful are graph neural networks?arXiv preprint arXiv:1810.00826(2018).
[39]
Runzhen Xue, Dengke Han, Mingyu Yan, Mo Zou, Xiaocheng Yang, Duo Wang, Wenming Li, Zhimin Tang, John Kim, Xiaochun Ye, and Dongrui Fan. 2024. HiHGNN: Accelerating HGNNs Through Parallelism and Data Reusability Exploitation. IEEE Transactions on Parallel and Distributed Systems 35, 7 (2024), 1122–1138.
[40]
Runzhen Xue, Mingyu Yan, Dengke Han, Yihan Teng, Zhimin Tang, Xiaochun Ye, and Dongrui Fan. 2024. GDR-HGNN: A Heterogeneous Graph Neural Networks Accelerator Frontend with Graph Decoupling and Recoupling. ArXiv abs/2404.04792(2024).
[41]
Mingyu Yan, Zhaodong Chen, Lei Deng, et al. 2020. Characterizing and understanding GCNs on GPU. IEEE Computer Architecture Letters 19, 1 (2020), 22–25.
[42]
Mingyu Yan, Xing Hu, Shuangchen Li, et al. 2019. Alleviating Irregularity in Graph Analytics Acceleration: A Hardware/Software Co-Design Approach. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (Columbus, OH, USA) (MICRO ’52). Association for Computing Machinery, New York, NY, USA, 615–628.
[43]
Mingyu Yan, Mo Zou, Xiaocheng Yang, et al. 2022. Characterizing and Understanding HGNNs on GPUs. IEEE Computer Architecture Letters 21, 2 (2022), 69–72.
[44]
Xiaocheng Yang, Mingyu Yan, Shirui Pan, Xiaochun Ye, and Dongrui Fan. 2023. Simple and Efficient Heterogeneous Graph Neural Network. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37.
[45]
Yanfang Ye, Shifu Hou, Lingwei Chen, Jingwei Lei, Wenqiang Wan, Jiabin Wang, Qi Xiong, and Fudong Shao. 2019. Out-of-sample node representation learning for heterogeneous graph in real-time android malware detection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (Macao, China) (IJCAI’19). AAAI Press, 4150–4156.
[46]
Hengrui Zhang, Zhongming Yu, Guohao Dai, Guyue Huang, Yufei Ding, Yuan Xie, and Yu Wang. 2022. Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective. In Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa Clara, CA, USA, August 29 - September 1, 2022, Diana Marculescu, Yuejie Chi, and Carole-Jean Wu (Eds.). mlsys.org.
[47]
Hengrui Zhang, Zhongming Yu, Guohao Dai, Guyue Huang, Yufei Ding, Yuan Xie, and Yu Wang. 2022. Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspective. In Proceedings of the Fifth Conference on Machine Learning and Systems, MLSys 2022, Santa Clara, CA, USA, August 29 - September 1, 2022, Diana Marculescu, Yuejie Chi, and Carole-Jean Wu (Eds.). mlsys.org.
[48]
Zhihui Zhang, Jingwen Leng, Lingxiao Ma, Youshan Miao, Chao Li, and Minyi Guo. 2020. Architectural implications of graph neural networks. IEEE Computer architecture letters 19, 1 (2020), 59–62.
[49]
Anping Zhao and Yu Yu. 2021. Context Aware Sentiment Link Prediction in Heterogeneous Social Network. Cognitive Computation 14(2021), 300 – 309.
[50]
Da Zheng, Xiang Song, Chengru Yang, Dominique LaSalle, and George Karypis. 2022. Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Heterogeneous Graphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Washington DC, USA) (KDD ’22). Association for Computing Machinery, New York, NY, USA, 4582–4591.
[51]
Weida Zhong, Qiuling Suo, Xiaowei Jia, Aidong Zhang, and Lu Su. 2021. Heterogeneous Spatio-Temporal Graph Convolution Network for Traffic Forecasting with Missing Values. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). 707–717.
[52]
Jie Zhou, Ganqu Cui, Shengding Hu, et al. 2020. Graph neural networks: A review of methods and applications. AI Open 1(2020), 57–81.
Index Terms
- Characterizing and Understanding HGNN Training on GPUs
Recommendations
Characterizing and Understanding Distributed GNN Training on GPUs
Graph neural network (GNN) has been demonstrated to be a powerful model in many domains for its effectiveness in learning over graphs. To scale GNN training for large graphs, a widely adopted approach is distributed training which accelerates training ...
Semi-supervised heterogeneous graph contrastive learning with label-guided
AbstractHeterogeneous Graph Neural Networks represent a powerful approach to understand and utilize the intricate structures and semantics within complex graphs. When it comes to semi-supervised learning on graphs, the challenge lies in effectively ...
Representation characterizations of chordal bipartite graphs
A bipartite graph is chordal bipartite if it does not contain an induced cycle of length at least six. We give three representation characterizations of chordal bipartite graphs. More precisely, we show that a bipartite graph is chordal bipartite if and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
EISSN:1544-3973
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s).
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 04 November 2024
Accepted: 23 October 2024
Revised: 19 September 2024
Received: 16 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 90Total Downloads
- Downloads (Last 12 months)90
- Downloads (Last 6 weeks)90
Reflects downloads up to 10 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in