More Web Proxy on the site http://driver.im/

research-article

Open access

152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulation

Authors:

Tsuyoshi Ichimura,

Kentaro Koyama,

Ryota Kusakabe,

Lalith Maddegedara,

Tatsuo Nishiki,

Seiya Nishizawa,

Naonori UedaAuthors Info & Claims

HPCAsia '22: International Conference on High Performance Computing in Asia-Pacific Region

Pages 18 - 29

https://doi.org/10.1145/3492805.3492814

Published: 07 January 2022 Publication History

All formats PDF

Abstract

We have used data learning and low-precision computation to develop an implicit solver that demonstrates high performance up to 152,352 computer nodes (609,408 MPI processes × 12 OpenMP threads = 7,312,896 parallel computation) and conducted an unprecedented ultra-large-scale analysis of ultra-high-fidelity fault-structure systems using nonlinear dynamic finite element analysis on three-dimensional low-order unstructured elements. The developed solver achieved 25.45-fold speedup from the state of the art solver on Fugaku and attained weak scaling efficiency of 93.7% from 9.391 billion DOF@578 computer nodes to 1.201 trillion DOF@73,984 computer nodes on performance measurement problems. Moreover, a realistic 324 billion DOF application example, which is difficult to obtain performance for, was computed in high performance. Since the developed solver is based on a highly generalizable algorithm, it is expected to contribute not only to earthquake simulation on Fugaku but also to the enhancement of similar applications in other fields and on other supercomputers.

References

[1]

[1] T.H. Jordan, “Societal Impact of Earthquake Simulations at Extreme Scale,” SC ’15 Invited Talk Spotlight.

[2]

[2] L. Carrington, D. Komatitsch, M. Laurenzano, M. Tikir, D. Michea, N.L. Goff, A. Snavely, and J. Tromp, “High-frequency simulations of global seismic wave propagation using SPECFEM3DGLOBE,” Proceedings of SC ’08, pp. 1–11, 2008.

[3]

[3] M. Rietmann, P. Messmer, T. Nissen-Meyer, D. Peter, P. Basini, D. Komatitsch, O. Schenk, J. Tromp, L. Boschi, and D. Giardini, “Forward and adjoint simulations of seismic wave propagation on emerging largescale GPU architectures,” Proceedings of SC ’12, pp. 1–11, 2012.

[4]

[4] Y. Cui, E. Poyraz, K. B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. M. Day, P. J. Maechling, and T. H. Jordan, “Physics-based seismic hazard analysis on petascale heterogeneous supercomputers,” Proceedings of SC ’13, pp. 1–12, 2013.

Digital Library

[5]

[5] A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A-A. Gabriel, C. Pelties, A. Bode, W. Barth, X-K. Liao, K. Vaidyanathan, M. Smelyanskiy, and P. Dubey, “Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers,” Proceedings of SC ’14, pp. 3–14, 2014.

Digital Library

[6]

[6] D. Roten, Y. Cui, K.B. Olsen, S.M. Day, K. Withers, W.H. Savran, P. Wang, and D. Mu, “High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers,” Proceedings of SC ’16, pp. 957–968, 2016.

[7]

[7] H. Fu, C. He, B. Chen, Z. Yin, Z. Zhang, W. Zhang, T. Zhang, W. Xue, W. Liu, W. Yin, G. Yang, and X. Chen, “15-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of Realistic 10 Hz Scenarios,” Proceedings of SC ’17, article No. 2, 2017.

[8]

[8] C. Uphoff, S. Rettenberger, M. Bader, E. H. Madden, T. Ulrich, S. Wollherr, and A.-A. Gabriel, “Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake,” Proceedings of SC ’17, pp. 1–16, 2017.

Digital Library

[9]

[9] T. Ichimura, K. Fujita, S. Tanaka, M. Hori, M. Lalith, Y. Shizawa, and H. Kobayashi. “Physics-based urban earthquake simulation enhanced by 10.7 BlnDOF x 30 K time-step unstructured FE non-linear seismic wave simulation,” Proceedings of SC ’14, pp. 15–26, 2014.

Digital Library

[10]

[10] T. Ichimura, K. Fujita, P. E. B. Quinay, L. Maddegedara, M. Hori, S. Tanaka, Y. Shizawa, H. Kobayashi, and K. Minami, “Implicit Nonlinear Wave Simulation with 1.08T DOF and 0.270T Unstructured Finite Elements to Enhance Comprehensive Earthquake Simulation,” Proceedings of SC ’15, pp. 1–12, 2015.

Digital Library

[11]

[11] T. Ichimura, K. Fujita, T. Yamaguchi, A. Naruse, J. C. Wells, T. C. Schulthess, T. P. Straatsma, C. J. Zimmer, M. Martinasso, K. Nakajima, M. Hori, and L. Maddegedara, “A fast scalable implicit solver for nonlinear time-evolution earthquake city problem on low-ordered unstructured finite elements with artificial intelligence and transprecision computing,” Proceedings of SC ’18, article No. 49, 2018.

Digital Library

[12]

[12] N. M. Newmark, “A method of computation for structural dynamics,” Journal of the Engineering Mechanics Division, 85 (EM3), pp. 67–94, 1959.

[13]

[13] Y.D. Isbiliroglu, R. Taborda, and J. Bielak, “Coupled soil-structure interaction effects of building clusters during earthquakes,” Earthq. Spectra, 31, pp. 463–500, 2013.

[14]

[14] W. Zhang, D. Restrepo, J. G.F. Crempien, B. Erkmen, R. Taborda, A. Kurtulus, and E. Taciroglu, “A computational workflow for rupture-to-structural-response simulation and its application to Istanbul,” Earthquake Engineering Structural Dynamics, 50, pp. 177–196, 2021.

[15]

[15] T. Ichimura, M. Hori, P. E. B. Quinay, L. Maddegedara, T. Suzuki, and S. Noguchi, “Comprehensive numerical analysis of fault-structure systems – Computation of the large-scale seismic structural response to a given earthquake scenario –,” Earthquake Engineering & Structural Dynamics, 41, pp 795–811, 2012.

[16]

[16] The American Society of Mechanical Engineers, Guide for verification and validation in computational solid mechanics, 2006.

[17]

[17] T. Ichimura, M. Hori, and J. Bielak, “A Hybrid Multiresolution Meshing Technique for Finite Element Three-Dimensional Earthquake Ground Motion Modeling in Basins Including Topography,” Geophysical Journal International, 177, pp 1221–1232, 2009.

[18]

[18] K. Fujita, K. Katsushima, T. Ichimura, M. Horikoshi, K. Nakajima, M. Hori, and L. Maddegedara, “Wave propagation simulation of complex multi-material problems with fast low-order unstructured finite-element meshing and analysis,” Proceedings of the International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia 2018), pp. 24–35, 2018.

Digital Library

[19]

[19] GEBCO Bathymetric Compilation Group 2020, The GEBCO_2020 Grid - a continuous terrain model of the global oceans and land, British Oceanographic Data Centre, National Oceanography Centre, 2020.

[20]

[20] K. Fujita, K. Koyama, K. Minami, H. Inoue, S. Nishizawa, M. Tsuji, T. Nishiki, T. Ichimura, M. Hori, and L. Maddegedara, “High-fidelity nonlinear low-order unstructured implicit finite-element seismic simulation of important structures by accelerated element-by-element method,” Journal of Computational Science, 49, article No. 101277, 2021.

[21]

[21] T. Ichimura, K. Fujita, K. Koyama, R. Kusakabe, K. Minami, H. Inoue, S. Nishizawa, M. Tsuji, T. Nishiki, M. Hori, L. Maddegedara, and N. Ueda, “Fast Scalable Implicit Solver with Convergence of Physics-Based Simulation and Data-Driven Learning,” Research Poster for SC ’20, 2020.

[22]

[22] J.M. Winget and T.J. Hughes, “Solution algorithms for nonlinear transient heat conduction analysis employing element-by-element iterative strategies,” Computer Methods in Applied Mechanics and Engineering, 52, pp. 711–815, 1985.

[23]

[23] “EBE kernel for A64FX CPUs [Online].” To be available.

[24]

[24] J. N. Kutz, S. L. Brunton, B. W. Brunton, and J. L. Proctor, Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems, Society for Industrial Math Applied Mathematic, 2016.

[25]

[25] G. H. Golub and Q. Ye, “Inexact preconditioned conjugate gradient method with inner-outer iteration,” SIAM Journal on Scientific Computing, 21, pp. 1305–1320, 1999.

Digital Library

[26]

[26] M. Kubale, Graph Colorings, American Mathematical Society, 2004.

[27]

[27] C. Cecka, A. Lew, E. Darve, “Assembly of finite element methods on graphics processors,” Int. J. Numer. Meth. Eng., 85, pp. 640–669, 2011.

[28]

[28] G. Karypis and V. Kumar, “A fast and high quality multilevel scheme for partitioning irregular graphs,” SIAM Journal on scientific Computing, 20, pp. 359–392, 1998.

Digital Library

[29]

[29] “Strong ground motion of The Southern Hyogo prefecture earthquake in 1995 observed at Kobe JMA observatory, Japan Meteorological Agency, [Online].” https://www.data.jma.go.jp/svd/eqev/data/kyoshin/jishin/

[30]

[30] I.M. Idriss, R. Dobry, and R.D. Sing, “Nonlinear behavior of soft clays during cyclic loading,” J. Geotech. Eng. Div, 104, pp. 1427-1447, 1978.

[31]

[31] G. Masing, “Eigenspannungen und Verfestigung beim Messing,” Proceedings of the 2nd International Congress of Applied Mechanics, pp. 332–335, 1926.

[32]

[32] T. Yoshida, “Fujitsu High Performance CPU for the Post-K Computer,” IEEE Hot Chips: A Symposium on High Performance Chips, 2018.

[33]

[33] “Supercomputer Fugaku, RIKEN Center for Computational Science [Online].” https://www.r-ccs.riken.jp/en/fugaku/

[34]

[34] M. Sato, Y. Ishikawa, H. Tomita, Y. Kodama, T. Odajima, M. Tsuji, H. Yashiro, M. Aoki, N. Shida, I. Miyoshi, K. Hirai, A. Furuya, A. Asato, K. Morita, and T. Shimizu, “Co-design for A64FX manycore processor and ”Fugaku”,” Proceedings of SC ’20, pp. 1–15, 2020.

[35]

[35] Y. Ajima, T. Kawashima, T. Okamoto, N. Shida, K. Hirai, T. Shimizu, S. Hiramoto, Y. Ikeda, T. Yoshikawa, K. Uchida, and T. Inoue, “The Tofu Interconnect D,” 2018 IEEE International Conference on Cluster Computing (CLUSTER), pp. 646–654, 2018.

[36]

[36] “Oakbridge-CX Supercomputer System, Information Technology Center, The University of Tokyo [Online].” https://www.cc.u-tokyo.ac.jp/en/supercomputer/obcx/service/

Cited By

Otsuka YTamari YFujita KIchimura T(2024)Large-scale 3D seismic response analysis considering soil liquefaction in an urban area using the supercomputer “Fugaku”Japanese Geotechnical Society Special Publication10.3208/jgssp.v10.OS-35-0610:46(1735-1740)Online publication date: 2024
https://doi.org/10.3208/jgssp.v10.OS-35-06
OTSUKA YTAMARI YFUJITA KICHIMURA T(2024)HAZARD MAP FOR LIQUEFACTION BY 3D SEISMIC RESPONSE ANALYSIS USING THE SUPERCOMPUTER “FUGAKU”スーパーコンピュータ「富岳」を用いた3次元地震応答解析による液状化のハザードマップ作成の提案Japanese Journal of JSCE10.2208/jscejj.23-1310780:13(n/a)Online publication date: 2024
https://doi.org/10.2208/jscejj.23-13107
Gill ALalith MHori MOgawa Y(2024)Analysis of postdisaster economy using high‐resolution disaster and economy simulationsRisk Analysis10.1111/risa.17662Online publication date: 9-Oct-2024
https://doi.org/10.1111/risa.17662
Show More Cited By

Index Terms

152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulation

Index terms have been assigned to the content through auto-classification.

Recommendations

The Effect of Time Constraints on Scaled Speedup

Gustafson, Montry, and Benner introduced the concept of scaled speedup to characterize the capabilities of distributed-memory multiprocessors. They argued that, for a fixed-size problem, the behavior of the speedup of an algorithm as a function of the ...
Parallel Multithreaded Satisfiability Solver

We describe the design and implementation of a highly optimized, multithreaded algorithm for the propositional satisfiability problem. The algorithm is based on the Davis-Putnam-Logemann-Loveland sequential algorithm, but includes many of the ...
A portable and scalable multigrid solver for nonlinear structural mechanics problems

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

HPCAsia '22: International Conference on High Performance Computing in Asia-Pacific Region

January 2022

145 pages

ISBN:9781450384988

DOI:10.1145/3492805

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 January 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Ministry of Education, Culture, Sports, Science and Technology

Conference

HPC Asia2022

HPC Asia2022: International Conference on High Performance Computing in Asia-Pacific Region

January 12 - 14, 2022

Virtual Event, Japan

Acceptance Rates

Overall Acceptance Rate 69 of 143 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
1,125
Total Downloads

Downloads (Last 12 months)355
Downloads (Last 6 weeks)69

Reflects downloads up to 20 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Otsuka YTamari YFujita KIchimura T(2024)Large-scale 3D seismic response analysis considering soil liquefaction in an urban area using the supercomputer “Fugaku”Japanese Geotechnical Society Special Publication10.3208/jgssp.v10.OS-35-0610:46(1735-1740)Online publication date: 2024
https://doi.org/10.3208/jgssp.v10.OS-35-06
OTSUKA YTAMARI YFUJITA KICHIMURA T(2024)HAZARD MAP FOR LIQUEFACTION BY 3D SEISMIC RESPONSE ANALYSIS USING THE SUPERCOMPUTER “FUGAKU”スーパーコンピュータ「富岳」を用いた3次元地震応答解析による液状化のハザードマップ作成の提案Japanese Journal of JSCE10.2208/jscejj.23-1310780:13(n/a)Online publication date: 2024
https://doi.org/10.2208/jscejj.23-13107
Gill ALalith MHori MOgawa Y(2024)Analysis of postdisaster economy using high‐resolution disaster and economy simulationsRisk Analysis10.1111/risa.17662Online publication date: 9-Oct-2024
https://doi.org/10.1111/risa.17662
Yuan MLiu QGan LYang G(2024)ESFLOW: Mapping Large-Scale Earthquake Simulation to Spatial Computing Systems2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558278(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558278
Wan WGan LWang WYin ZTian HZhang ZWang YHua MLiu XXiang SHe ZWang ZGao PDuan XLiu WXue WFu HYang GChen XSong ZChen YLiu XZhang WMohror KArnold DBadia R(2023)69.7-PFlops Extreme Scale Earthquake Simulation with Crossing Multi-faults and Topography on SunwayProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613209(1-15)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3613209
Murakami SFujita KIchimura THori THori MLalith MUeda N(2023)Development of 3D Viscoelastic Crustal Deformation Analysis Solver with Data-Driven Method on GPUComputational Science – ICCS 202310.1007/978-3-031-36021-3_45(423-437)Online publication date: 3-Jul-2023
https://dl.acm.org/doi/10.1007/978-3-031-36021-3_45
Touhami SGatti FLopez-Caballero FCottereau Rde Abreu Corrêa LAubry LClouteau D(2022)SEM3D: A 3D High-Fidelity Numerical Earthquake Simulator for Broadband (0–10 Hz) Seismic Response Prediction at a Regional ScaleGeosciences10.3390/geosciences1203011212:3(112)Online publication date: 2-Mar-2022
https://doi.org/10.3390/geosciences12030112
Fujita KMurakami SIchimura THori THori MLalith MUeda N(2022)Scalable Finite-Element Viscoelastic Crustal Deformation Analysis Accelerated with Data-Driven Method2022 IEEE/ACM Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH)10.1109/ScalAH56622.2022.00008(18-25)Online publication date: Nov-2022
https://doi.org/10.1109/ScalAH56622.2022.00008

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents