A Performance-Portable Kilometer-Scale Global Ocean Model on ORISE and New Sunway Heterogeneous Supercomputers
Authors: 
Junlin Wei, Xiang Han, Jiangfeng Yu, Jinrong Jiang, Hailong Liu, Pengfei Lin, Maoxue Yu, Kai Xu, Lian Zhao, Pengfei Wang, Weipeng Zheng, Jingwei Xie, Yanzhi Zhou, Tao Zhang, Feng Zhang, Yehong Zhang, Yue Yu, Yuzhu Wang, Yidi Bai, Chen Li, Zipeng Yu, Haoyu Deng, Yaxin Li, Xuebin ChiAuthors Info & Claims
Article No.: 3, Pages 1 - 12
Abstract
Ocean general circulation models (OGCMs) are indispensable for studying the multi-scale oceanic processes and climate change. High-resolution ocean simulations require immense computational power and thus become a challenge in climate science. We present LICOMK++, a performance-portable OGCM using Kokkos, to facilitate global kilometer-scale ocean simulations. The breakthroughs include: (1) we enhance cutting-edge Kokkos with the Sunway architecture, enabling LICOMK++ to become the first performance-portable OGCM on diversified architectures, i.e., Sunway processors, CUDA/HIP-based GPUs, and ARM CPUs. (2) LICOMK++ overcomes the one simulated-years-per-day (SYPD) performance challenge for global realistic OGCM at 1-km resolution. It records 1.05 and 1.70 SYPD with a parallel efficiency of 54.8% and 55.6% scaling on almost the entire new Sunway supercomputer and two-thirds of the ORISE supercomputer. (3) LICOMK++ is the first global 1-km-resolution realistic OGCM to generate scientific results. It successfully reproduces mesoscale and submesoscale structures that have considerable climate effects.
References
[1]
V. Masson-Delmotte, P. Zhai, A. Pirani, S. L. Connors, C. Péan, S. Berger, N. Caud, Y. Chen, L. Goldfarb, M. Gomis et al., "Climate change 2021: the physical science basis," Contribution of working group I to the sixth assessment report of the intergovernmental panel on climate change, vol. 2, no. 1, p. 2391, 2021.
[2]
Z. Li, M. H. England, and S. Groeskamp, "Recent acceleration in global ocean heat accumulation by mode and intermediate waters," Nature Communications, vol. 14, no. 1, p. 6888, 2023.
[3]
Z. Su, J. Wang, P. Klein, A. F. Thompson, and D. Menemenlis, "Ocean submesoscales as a key component of the global heat budget," Nature communications, vol. 9, no. 1, p. 775, 2018.
[4]
N. Beech, T. Rackow, T. Semmler, S. Danilov, Q. Wang, and T. Jung, "Long-term evolution of ocean eddy activity in a warming world," Nature climate change, vol. 12, no. 10, pp. 910--917, 2022.
[5]
S. Wang, Z. Jing, L. Wu, W. Cai, P. Chang, H. Wang, T. Geng, G. Danabasoglu, Z. Chen, X. Ma et al., "El Niño/Southern Oscillation inhibited by submesoscale ocean eddies," Nature Geoscience, vol. 15, no. 2, pp. 112--117, 2022.
[6]
H. T. Hewitt, M. Roberts, P. Mathiot, A. Biastoch, E. Blockley, E. P. Chassignet, B. Fox-Kemper, P. Hyder, D. P. Marshall, E. Popova et al., "Resolving and parameterising the ocean mesoscale in earth system models," Current Climate Change Reports, vol. 6, pp. 137--152, 2020.
[7]
D. B. Chelton, M. G. Schlax, and R. M. Samelson, "Global observations of nonlinear mesoscale eddies," Progress in oceanography, vol. 91, no. 2, pp. 167--216, 2011.
[8]
K. Stewart, A. M. Hogg, S. Griffies, A. Heerdegen, M. Ward, P. Spence, and M. H. England, "Vertical resolution of baroclinic modes in global ocean models," Ocean Modelling, vol. 113, pp. 50--65, 2017.
[9]
R. Hallberg, "Using a resolution function to regulate parameterizations of oceanic mesoscale eddy effects," Ocean Modelling, vol. 72, pp. 92--103, 2013.
[10]
P. Bauer, P. D. Dueben, T. Hoefler, T. Quintino, T. C. Schulthess, and N. P. Wedi, "The digital revolution of Earth-system science," Nature Computational Science, vol. 1, no. 2, pp. 104--113, 2021.
[11]
C. R. Trott, D. Lebrun-Grandié, D. Arndt, J. Ciesko, V. Dang, N. Ellingwood, R. Gayatri, E. Harvey, D. S. Hollman, D. Ibanez et al., "Kokkos 3: Programming model extensions for the exascale era," IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 4, pp. 805--817, 2021.
[12]
H. C. Edwards, C. R. Trott, and D. Sunderland, "Kokkos: Enabling manycore performance portability through polymorphic memory access patterns," Journal of parallel and distributed computing, vol. 74, no. 12, pp. 3202--3216, 2014.
[13]
T. M. Evans, A. Siegel, E. W. Draeger, J. Deslippe, M. M. Francois, T. C. Germann, W. E. Hart, and D. F. Martin, "A survey of software implementations used by application codes in the Exascale Computing Project," The International Journal of High Performance Computing Applications, vol. 36, no. 1, pp. 5--12, 2022.
[14]
M. Taylor, P. M. Caldwell, L. Bertagna, C. Clevenger, A. Donahue, J. Foucar, O. Guba, B. Hillman, N. Keen, J. Krishna et al., "The simple cloud-resolving E3SM atmosphere model running on the Frontier exascale system," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2023, pp. 1--11.
[15]
A. S. Donahue, P. M. Caldwell, L. Bertagna, H. Beydoun, P. A. Bogenschutz, A. Bradley, T. C. Clevenger, J. G. Foucar, J.-C. Golaz, O. Guba et al., "To exascale and beyond-the simple cloud-resolving E3SM atmosphere model (SCREAM), a performance portable global atmosphere model for cloud-resolving scales," Authorea Preprints, 2024.
[16]
A. Khan, H. Sim, S. S. Vazhkudai, A. R. Butt, and Y. Kim, "An analysis of system balance and architectural trends based on top500 supercomputers," in The International Conference on High Performance Computing in Asia-Pacific Region, 2021, pp. 11--22.
[17]
W. Wan, L. Gan, W. Wang, Z. Yin, H. Tian, Z. Zhang, Y. Wang, M. Hua, X. Liu, S. Xiang et al., "7-pflops extreme scale earthquake simulation with crossing multi-faults and topography on Sunway," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2023, pp. 1--15.
[18]
S. Huang, J. Chen, Z. Zhang, X. Hao, J. Gu, H. An, C. Zhao, Y. Hu, Z. Wang, L. Chen et al., "Establishing a modeling system in 3-km horizontal resolution for global atmospheric circulation triggered by submarine volcanic eruptions with 400 billion smoothed particle hydrodynamics," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2023, pp. 1--12.
[19]
W. Hu, H. An, Z. Guo, Q. Jiang, X. Qin, J. Chen, W. Jia, C. Yang, Z. Luo, J. Li et al., "2.5 million-atom ab initio electronic-structure simulation of complex metallic heterostructures with DGDFT," in SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2022, pp. 1--13.
[20]
Z. Xuehong and L. Xinzhong, "A numerical world ocean general circulation model," Advances in atmospheric sciences, vol. 6, no. 1, pp. 44--61, 1989.
[21]
Y. Yu, S. Tang, H. Liu, P. Lin, and X. Li, "Development and evaluation of the dynamic framework of an ocean general circulation model with arbitrary orthogonal curvilinear coordinate," Chinese Journal of Atmospheric Sciences, vol. 42, no. 4, pp. 877--889, 2018.
[22]
Y. Li, H. Liu, M. Ding, P. Lin, Z. Yu, Y. Yu, Y. Meng, Y. Li, X. Jian, J. Jiang et al., "Eddy-resolving simulation of CAS-LICOM3 for phase 2 of the ocean model intercomparison project," Advances in Atmospheric Sciences, vol. 37, pp. 1067--1080, 2020.
[23]
P. Lin, Z. Yu, H. Liu, Y. Yu, Y. Li, J. Jiang, W. Xue, K. Chen, Q. Yang, B. Zhao et al., "LICOM model datasets for the CMIP6 ocean model intercomparison project," Advances in Atmospheric Sciences, vol. 37, pp. 239--249, 2020.
[24]
H. Tsujino, L. S. Urakawa, S. M. Griffies, G. Danabasoglu, A. J. Adcroft, A. E. Amaral, T. Arsouze, M. Bentsen, R. Bernardello, C. W. Böning et al., "Evaluation of global ocean-sea-ice model simulations based on the experimental protocols of the ocean model intercomparison project phase 2 (OMIP-2)," Geoscientific Model Development, vol. 13, no. 8, pp. 3643--3708, 2020.
[25]
E. P. Chassignet, S. G. Yeager, B. Fox-Kemper, A. Bozec, F. Castruccio, G. Danabasoglu, W. M. Kim, N. Koldunov, Y. Li, P. Lin et al., "Impact of horizontal resolution on global ocean-sea-ice model simulations based on the experimental protocols of the ocean model intercomparison project phase 2 (OMIP-2)," Geoscientific Model Development Discussions, vol. 2020, pp. 1--58, 2020.
[26]
A. M. Treguier, C. de Boyer Montégut, A. Bozec, E. P. Chassignet, B. Fox-Kemper, A. McC. Hogg, D. Iovino, A. E. Kiss, J. Le Sommer, Y. Li et al., "The mixed-layer depth in the ocean model intercomparison project (OMIP): impact of resolving mesoscale eddies," Geoscientific Model Development, vol. 16, no. 13, pp. 3849--3872, 2023.
[27]
Q. Wang, Q. Shu, A. Bozec, E. P. Chassignet, P. G. Fogli, B. Fox-Kemper, A. M. Hogg, D. Iovino, A. E. Kiss, N. Koldunov et al., "Impact of high resolution on arctic ocean simulations in ocean model intercomparison project phase 2 (OMIP-2)," Geoscientific Model Development Discussions, vol. 2023, pp. 1--46, 2023.
[28]
J. Mak, P. Choboter, and C. Lupo, "Numerical ocean modeling and simulation with CUDA," in OCEANS'11 MTS/IEEE KONA, 2011, pp. 1--6.
[29]
S. Xu, X. Huang, L.-Y. Oey, F. Xu, H. Fu, Y. Zhang, and G. Yang, "POM. gpu-v1. 0: a GPU-based princeton ocean model," Geoscientific Model Development, vol. 8, no. 9, pp. 2815--2827, 2015.
[30]
T. Aketh, S. Vadhiyar, P. Vinayachandran, and R. Nanjundiah, "High performance horizontal diffusion calculations in ocean models on Intel® Xeon Phi™ coprocessor systems," in 2016 IEEE 23rd International Conference on High Performance Computing (HiPC). IEEE, 2016, pp. 203--211.
[31]
T. Yamagishi and Y. Matsumura, "GPU acceleration of a non-hydrostatic ocean model with a multigrid Poisson/Helmholtz solver," Procedia Computer Science, vol. 80, pp. 1658--1669, 2016.
[32]
J. Jiang, P. Lin, J. Wang, H. Liu, X. Chi, H. Hao, Y. Wang, W. Wang, and L. Zhang, "Porting LASG/IAP climate system ocean model to GPUs using OpenAcc," IEEE Access, vol. 7, pp. 154490--154501, 2019.
[33]
B. Fox-Kemper, A. Adcroft, C. W. Böning, E. P. Chassignet, E. Curchitser, G. Danabasoglu, C. Eden, M. H. England, R. Gerdes, R. J. Greatbatch et al., "Challenges and prospects in ocean circulation models," Frontiers in Marine Science, vol. 6, p. 65, 2019.
[34]
Y. Zeng, L. Wang, J. Zhang, G. Zhu, Y. Zhuang, and Q. Guo, "Redistributing and optimizing high-resolution ocean model POP2 to million Sunway cores," in Algorithms and Architectures for Parallel Processing: 20th International Conference, ICA3PP 2020, New York City, NY, USA, October 2-4, 2020, Proceedings, Part I 20. Springer, 2020, pp. 275--289.
[35]
D. Häfner, R. Nuterman, and M. Jochum, "Fast, cheap, and turbulent---global ocean modeling with GPU acceleration in python," Journal of Advances in Modeling Earth Systems, vol. 13, no. 12, p. e2021MS002717, 2021.
[36]
Y. Ye, Z. Song, S. Zhou, Y. Liu, Q. Shu, B. Wang, W. Liu, F. Qiao, and L. Wang, "swNEMO_v4.0: an ocean model NEMO for the next generation Sunway supercomputer," Geoscientific Model Development Discussions, vol. 2022, pp. 1--29, 2022.
[37]
J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, "Julia: A fresh approach to numerical computing," SIAM Review, vol. 59, no. 1, pp. 65--98, Sep. 2017.
[38]
S. Silvestri, G. Wagner, C. Hill, M. R. Ardakani, J. Blaschke, J.-M. Campin, V. Churavy, N. Constantinou, A. Edelman, J. Marshall, A. Ramadhan, A. Souza, and R. Ferrari, "Oceananigans.jl: A model that achieves breakthrough resolution, memory and energy efficiency in global ocean simulations," 2023.
[39]
S. Silvestri, G. L. Wagner, N. C. Constantinou, C. N. Hill, J.-M. Campin, A. N. Souza, S. Bishnu, V. Churavy, J. C. Marshall, and R. Ferrari, "A GPU-based ocean dynamical core for routine mesoscale-resolving climate simulations," Authorea Preprints, 2024.
[40]
L. Bertagna, M. Deakin, O. Guba, D. Sunderland, A. M. Bradley, I. K. Tezaur, M. A. Taylor, and A. G. Salinger, "HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model," Geoscientific Model Development, vol. 12, no. 4, pp. 1423--1441, 2019.
[41]
L. Bertagna, O. Guba, M. A. Taylor, J. G. Foucar, J. Larkin, A. M. Bradley, S. Rajamanickam, and A. G. Salinger, "A performance-portable nonhydrostatic atmospheric dycore for the energy exascale earth system model running at cloud-resolving resolutions," in Sc20: International conference for high performance computing, networking, storage and analysis. IEEE, 2020, pp. 1--14.
[42]
J. Wei, J. Jiang, H. Liu, F. Zhang, P. Lin, P. Wang, Y. Yu, X. Chi, L. Zhao, M. Ding et al., "LICOM3-CUDA: A GPU version of LASG/IAP climate system ocean model version 3 based on CUDA," The Journal of Supercomputing, pp. 1--31, 2023.
[43]
P. Wang, J. Jiang, P. Lin, M. Ding, J. Wei, F. Zhang, L. Zhao, Y. Li, Z. Yu, W. Zheng et al., "The GPU version of LASG/IAP climate system ocean model version 3 (LICOM3) under the heterogeneous-compute interface for portability (HIP) framework and its large-scale application," Geoscientific Model Development, vol. 14, no. 5, pp. 2781--2799, 2021.
[44]
J. Wei, P. Lin, J. Jiang, H. Liu, L. Zhao, Y. Zhang, X. Han, F. Zhang, J. Huang, Y. Wang et al., "Accelerating LASG/IAP climate system ocean model version 3 for performance portability using Kokkos," Future Generation Computer Systems, 2024.
[45]
W.-Y. Huang, B. Wang, Y.-Q. Yu, and L.-J. Li, "Improvements in licom2. part i: Vertical mixing," Journal of Atmospheric and Oceanic Technology, vol. 31, no. 2, pp. 531--544, 2014.
[46]
V. Canuto, A. Howard, Y. Cheng, C. J. Muller, A. Leboissetier, and S. Jayne, "Ocean turbulence, iii: New giss vertical mixing scheme," Ocean Modelling, vol. 34, no. 3--4, pp. 70--91, 2010.
[47]
Y. Rucong, "A two---step shape---preserving advection scheme," Advances in Atmospheric Sciences, vol. 11, no. 4, pp. 479--490, 1994.
[48]
J. L. Gustafson, "Reevaluating amdahl's law," Communications of the ACM, vol. 31, no. 5, pp. 532--533, 1988.
[49]
R. Schubert, J. Gula, R. J. Greatbatch, B. Baschek, and A. Biastoch, "The submesoscale kinetic energy cascade: Mesoscale absorption of submesoscale mixed layer eddies and frontal downscale fluxes," Journal of Physical Oceanography, vol. 50, no. 9, pp. 2573--2589, 2020.
[50]
J. R. Taylor and A. F. Thompson, "Submesoscale dynamics in the upper ocean," Annual Review of Fluid Mechanics, vol. 55, no. 1, pp. 103--127, 2023.
[51]
J. Xie, X. Wang, H. Liu, P. Lin, J. Yu, and Z. Yu, "Isom 1.0: A fully mesoscale-resolving idealized southern ocean model and the diversity of multiscale eddy interactions," Geoscientific Model Development Discussions, vol. 2024, pp. 1--31, 2024.
[52]
J. McWilliams, "Submesoscale currents in the ocean," Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 472, no. 2189, p. 0117, 2016.
[53]
P. D. Dueben, N. Wedi, S. Saarinen, and C. Zeman, "Global simulations of the atmosphere at 1.45 km grid-spacing with the integrated forecasting system," Journal of the Meteorological Society of Japan. Ser. II, vol. 98, no. 3, pp. 551--572, 2020.
[54]
P. Neumann, P. Düben, P. Adamidis, P. Bauer, M. Brück, L. Kornblueh, D. Klocke, B. Stevens, N. Wedi, and J. Biercamp, "Assessing the scales in numerical weather and climate predictions: will exascale be the rescue?" Philosophical Transactions of the Royal Society A, vol. 377, no. 2142, p. 20180148, 2019.
Index Terms
- A Performance-Portable Kilometer-Scale Global Ocean Model on ORISE and New Sunway Heterogeneous Supercomputers
Index terms have been assigned to the content through auto-classification.
Recommendations
Accelerating LASG/IAP climate system ocean model version 3 for performance portability using Kokkos
AbstractIn this paper, the performance portability of the LASG/IAP Climate System Ocean Model version 3 (LICOM3) is demonstrated based on the C++ library Kokkos. Kokkos enables application execution in various High-Performance Computing (HPC) ...
Highlights- Adapting a global ocean model to new performance portability technologies.
- The performance-portability code is compared to different raw back-end codes.
- The proposed code achieving 3.4 Simulated Years Per Day at a resolution of 0 . ...
Towards Efficient SpMV on Sunway Manycore Architectures
ICS '18: Proceedings of the 2018 International Conference on SupercomputingSparse Matrix-Vector Multiplication (SpMV) is an essential computation kernel for many data-analytic workloads running in both supercomputers and data centers. The intrinsic irregularity in SpMV is challenging to achieve high performance, especially ...
LICOM3-CUDA: a GPU version of LASG/IAP climate system ocean model version 3 based on CUDA
AbstractThe ocean general circulation model (OGCM) is an essential tool for researching oceanography and atmospheric science. The LASG/IAP climate system ocean model version 3 (LICOM3) is a parallel version of the OGCM. Our goal is to implement and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2024
1758 pages
ISBN:9798350352917
Sponsors
Publisher
IEEE Press
Publication History
Published: 17 November 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SC '24
Sponsor:
SC '24: The International Conference for High Performance Computing, Networking, Storage, and Analysis
November 17 - 22, 2024
GA, Atlanta, USA
Acceptance Rates
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 145Total Downloads
- Downloads (Last 12 months)145
- Downloads (Last 6 weeks)145
Reflects downloads up to 22 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in