More Web Proxy on the site http://driver.im/

research-article

Open access

Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation

Authors:

Tsuyoshi Ichimura,

Pher Errol Balde Quinay,

Lalith Maddegedara,

Yoshihisa Shizawa,

Hiroshi Kobayashi,

Kazuo MinamiAuthors Info & Claims

SC '15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 4, Pages 1 - 12

https://doi.org/10.1145/2807591.2807674

Published: 15 November 2015 Publication History

Abstract

This paper presents a new heroic computing method for unstructured, low-order, finite-element, implicit nonlinear wave simulation: 1.97 PFLOPS (18.6% of peak) was attained on the full K computer when solving a 1.08T degrees-of-freedom (DOF) and 0.270T-element problem. This is 40.1 times more DOF and elements, a 2.68-fold improvement in peak performance, and 3.67 times faster in time-to-solution compared to the SC14 Gordon Bell finalist's state-of-the-art simulation. The method scales up to the full K computer with 663,552 CPU cores with 96.6% sizeup efficiency, enabling solving of a 1.08T DOF problem in 29.7 s per time step. Using such heroic computing, we solved a practical problem involving an area 23.7 times larger than the state-of-the-art, and conducted a comprehensive earthquake simulation by combining earthquake wave propagation analysis and evacuation analysis. Application at such scale is a groundbreaking accomplishment and is expected to change the quality of earthquake disaster estimation and contribute to society.

References

[1]

P. Somerville, N. Collins, N. Abrahamson, R. Graves, and C. Saikia, "GROUND MOTION ATTENUATION RELATIONS FOR THE CENTRAL AND EASTERN UNITED STATES," Final Report (June 30, 2001). {Online}. http://earthquake.usgs.gov/hazards/products/conterminous/2008/99HQGR0098.pdf

[2]

Second report of the Nankai Trough Large Earthquake Model Committee, Cabinet Office, Government of Japan (August 28,2012). {Online}. http://www.bousai.go.jp/jishin/nankai/model/index.html

[3]

Disaster assessment of Tokyo due to large earthquakes such as the Nankai Trough Earthquake, Tokyo Metropolitan Government (May 14, 2013). {Online}. http://www.bousai.metro.tokyo.jp/taisaku/1000902/1000402.html

[4]

H. Miyazaki, Y. Kusano, N. Shinjou, F. Shoji, M. Yokokawa and T. Watanabe. "Overview of the K computer system," FUJITSU Sci. Tech. J., 2012, Vol. 48, No. 3, pp.302--309.

[5]

T. Tiankai, Y. Hongfeng, L. Ramirez-Guzman, J. Bielak, O. Ghattas, M. Kwan-Liu, and D. R. O'Hallaron, "From mesh generation to scientific visualization: an end-to-end approach to parallel supercomputing," Proceedings of the 2006 ACM/IEEE conference on Supercomputing (SC'06). ACM, New York, NY, USA, 2006, Article 91. DOI=10.1145/1188455.1188551

Digital Library

[6]

C. Yifeng Cui, K. B. Olsen, T. H. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D. K. Panda, A. Chourasia, J. Levesque, S. M. Day and P. Maechling, "Scalable Earthquake Simulation on Petascale Supercomputers," Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC'10). IEEE Computer Society, Washington, DC, USA, 2010, pp.1--20. DOI=10.1109/SC.2010.45 http://dx.doi.org/10.1109/SC.2010.45

Digital Library

[7]

M. Rietmann, P. Messmer, T. Nissen-Meyer, D. Peter, P. Basini, D. Komatitsch, O. Schenk, J. Tromp, L. Boschi, and D. Giardini. "Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC'12). IEEE Computer Society Press, Los Alamitos, CA, USA, 2012, Article 38, 11 pages.

Digital Library

[8]

Y. Cui, E. Poyraz, K. B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. M. Day, P. J. Maechling and T. H. Jordan. "Physics-based seismic hazard analysis on petascale heterogeneous supercomputers," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'13). IEEE Computer Society Press, New York, NY, USA, 2013, Article 70, 12 pages.

Digital Library

[9]

A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A-A. Gabriel, C. Pelties, A. Bode, W. Barth, X-K. Liao, K. Vaidyanathan, M. Smelyanskiy, P. Dubey, "Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'14), pp 3--14, 2014.

Digital Library

[10]

Japan Seismic Hazard Information Station, National Research Institute for Earth Science and Disaster Prevention. {Online}. http://www.j-shis.bosai.go.jp/en/

[11]

Report of the Assumed Tokyo Large Earthquake Model Committee, Cabinet Office, Government of Japan (December 28, 2013). {Online}. http://www.bousai.go.jp/kaigirep/chuobou/senmon/shutochokkajishinmodel/

[12]

P. E. B. Quinay, T. Ichimura, M. Hori, A. Nishida, S. Yoshimura. "Seismic Structural Response Estimates of a High-Fidelity Fault-Structure System Model Using Multiscale Analysis with Parallel Simulation of Seismic Wave Propagation," Bulletin of the Seismological Society of America, 103, pp. 2094--2110 2013.

[13]

R. Taborda and J. Bielak. "Large-scale earthquake simulation: computational seismology and complex engineering systems," Computing in Science & Engineering, 13, pp. 14--27, 2011.

Digital Library

[14]

T. Ichimura, K. Fujita, M. Hori, T. Sakanoue and R. Hamanaka. "Three-dimensional nonlinear seismic ground response analysis of local site effects for estimating seismic behavior of buried pipelines," Journal of Pressure Vessel Technology, American Society of Mechanical Engineers, 136, paperID 041702, 2014.

[15]

K. Fujita, T. Ichimura, M. Hori, L. Maddegedara and S. Tanaka. "Scalable multicase urban earthquake simulation method for stochastic earthquake disaster estimation," Procedia Computer Science, 2015 (accepted).

[16]

T. Ichimura, K. Fujita, S. Tanaka, M. Hori, M. Lalith, Y. Shizawa, and H. Kobayashi. "Physics-based urban earthquake simulation enhanced by 10.7 BlnDOF x 30 K time-step unstructured FE non-linear seismic wave simulation," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'14), pp 15--26, 2014.

Digital Library

[17]

National Digital Soil Map, The Japanese Geotechincal Society, {Online}. http://www.denshi-jiban.jp/

[18]

Strong-motion seismograph networks (K-NET, KiK-net), National Research Institute for Earth Science and Disaster Prevention, {Online}. http://www.kyoshin.bosai.go.jp/

[19]

K. Zia, A. Riener, K. Farrahi and A. Ferscha. "A New Opportunity to Urban Evacuation Analysis: Very Large Scale Simulations of Social Agent Systems in Repast HPC," ACM/IEEE/SCS 26th Workshop on Principles of Advanced and Distributed Simulation, 2012.

Digital Library

[20]

M. L. L. Wijerathne, M. Hori, T. Ichimura, S. Tanaka. "Parallel Scalability Enhancements of Seismic Response and Evacuation Simulations of Integrated Earthquake Simulator," High Performance Computing for Computational Science - VECPAR 2012, Lecture Notes in Computer Science, Vol. 7851, 2013, pp 105--117, 2013.

[21]

M. L. Wijerathne, L. A. Melgar, M. Hori, T. Ichimura, and S. Tanaka. "HPC enhanced large urban area evacuation simulations with vision based autonomously navigating multiagents," Procedia Computer Science, pp 1515--1524, 2013.

[22]

L. E. A. Melgar, M. Lalith, M. Hori, T. Ichimura, S. Tanaka. "A Scalable Workbench for Large Urban Area Simulations, Comprised of Resources for Behavioural Models," Interactions and Dynamic Environments, PRIMA 2014: Principles and Practice of Multi-Agent Systems, Lecture Notes in Computer Science, Vol. 8861, pp 166--181, 2014.

[23]

I. M. Idriss, R. D. Singh and R. Dobry. "Nonlinear behavior of soft clays during cyclic loading," Journal of the Geotechnical Engineering Division, 1978, 104, pp. 1427--1447.

[24]

G. Masing. "Eigenspannungen und verfestigung beim messing," Proceedings of the 2nd International Congress of Applied Mechanics, 1926, pp. 332--335 (in German).

[25]

M. Ogino, R. Shioya and H. Kanayama. "An inexact balancing preconditioner for large-scale structural analysis," Journal of Conputational Science and Technology, 2008, 2(1), pp. 150--161.

[26]

H. Kawai, M. Ogino, R. Shioya and S. Yoshimura. "Large-scale elastplastic analysis using domain decomposition method optimized for multi-core CPU architecture," Key Engineering Materials, 2011, 462--463, pp. 605--610.

[27]

H. Akiba, T. Ohyama, Y. Shibata, K. Yuyama, Y. Katai, R. Takeuchi, T. Hoshino, S. Yoshimura, H. Noguchi, M. Gupta, J. Gunnels, V. Austel, Y. Sabharwal, R. Garg, S. Kato, T. Kawakami, S. Todokoro, J. Ikeda. "Large scale drop impact analysis of mobile phone using ADVC on Blue Gene/L," Proceedings of the 2006 ACM/IEEE conference on Supercomputing (SC'06), 2006, ACM, New York, NY, USA, Article 46. DOI=10.1145/1188455.1188503

Digital Library

[28]

J. Mandel, "Balancing domain decomposition," Communications in numerical methods in engineering, 1993, Volume 9, Issue 3, pp.233--241.

[29]

Earth Simulator (ES), http://www.jamstec.go.jp/es/en/es1/index.html

[30]

H. Bao, J. Bielak, O. Ghattas, D. R. O'Hallaron, L. F. Kallivokas, J. R. Shewchuk, J. Xu, "Earthquake ground motion modeling on parallel computers," in Proceedings of the 1996 ACM/IEEE conference on Supercomputing, 1996.

Digital Library

[31]

T. Ichimura, M. Hori, H. Kuwamoto, Earthquake Motion Simulation with Multi-Scale Finite Element Analysis on Hybrid Grid, Bulletin of the Seismological Society of America, 97, pp. 1133--1143, 2007.

[32]

T. Ichimura, M. Hori, J. Bielak. "A Hybrid multiresolution meshing technique for finite element three-dimensional earthquake ground motion modeling in basins including topography," Geophysical Journal International, 2009, 177, pp. 1221--1232.

[33]

G. H. Golub, Q. Ye Inexact conjugate gradient method with inner-outer iteration. SIAM., Journal on Scientific Computing 1997; 21(4):1305--1320.

Digital Library

[34]

Y. Saad, 2003, Iterative methods for sparse linear systems (2nd ed.), SIAM.

Digital Library

[35]

J. M. Winget, T. J. R. Hughes "Solution algorithms for nonlinear transient heat conduction analysis employing element-by- element iterative strategies," Computer Methods in Applied Mechanics and Engineering 1985; 52:711--815.

[36]

O. C. Zienkiewicz, R. L. Taylor. The Finite Element Method for Solid and Structural Mechanics (6th ed.), 2005, Elsevier.

[37]

METIS 5.1.0, {Online}. http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

[38]

E. Hairer, S. P. Nørsett, G. Wanner, 1993, Solving ordinary differential equations I: Non stiff problems (2nd ed.), Springer.

Digital Library

[39]

Y. Ajima, T. Inoue, S. Hiramoto and T. Shimizu. "Tofu: interconnect for the K computer," FUJITSU Sci. Tech. J., 2012, Vol.48, No.3, pp.280--285.

[40]

OpenMPI, {Online}. http://www.open-mpi.org/

[41]

Strong ground motion of The Southern Hyogo prefecture earthquake in 1995 observed at Kobe JMA observatory, Japan Meteorological Agency, {Online}. http://www.data.jma.go.jp/svd/eqev/data/kyoshin/jishin/hyogo nanbu/dat/H1171931.csv

Cited By

Zhang WDong YCrempien JArduino PKurtulus ATaciroglu E(2024)A comparison of ground motions predicted through one-dimensional site response analyses and three-dimensional wave propagation simulations at regional scalesEarthquake Spectra10.1177/8755293024123193540:2(1215-1234)Online publication date: 28-Feb-2024
https://doi.org/10.1177/87552930241231935
Gill ALalith MHori MOgawa Y(2024)Analysis of postdisaster economy using high‐resolution disaster and economy simulationsRisk Analysis10.1111/risa.17662Online publication date: 9-Oct-2024
https://doi.org/10.1111/risa.17662
Wan JWang WZhang Z(2024)Enhancing computational efficiency in 3-D seismic modelling with half-precision floating-point numbers based on the curvilinear grid finite-difference methodGeophysical Journal International10.1093/gji/ggae235238:3(1595-1611)Online publication date: 5-Jul-2024
https://doi.org/10.1093/gji/ggae235
Show More Cited By

Index Terms

Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation

Recommendations

18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

This paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (...
Physics-based urban earthquake simulation enhanced by 10.7 BlnDOF × 30 K time-step unstructured FE non-linear seismic wave simulation
SC '14: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

With the aim of dramatically improving the reliability of urban earthquake response analyses, we developed an unstructured 3-D finite-element-based MPI-OpenMP hybrid seismic wave amplification simulation code, GAMERA. On the K computer, GAMERA was able ...
Acceleration of element-by-element kernel in unstructured implicit low-order finite-element earthquake simulation using OpenACC on pascal GPUs
WACCPD '16: Proceedings of the Third International Workshop on Accelerator Programming Using Directives

The element-by-element computation used in matrix-vector multiplications is the key kernel for attaining high-performance in unstructured implicit low-order finite-element earthquake simulations. We accelerate this CPU-based element-by-element kernel by ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '15: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2015

985 pages

ISBN:9781450337236

DOI:10.1145/2807591

General Chair:
Jackie Kern
University of Illinois at Urbana-Champaign, Urbana, Illinois
,
Program Chair:
Jeffrey S. Vetter
Oak Ridge National Laboratory and Georgia Institute of Technology, Oak Ridge, Tennessee

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SC15

Sponsor:

SIGHPC
SIGARCH
IEEE-CS

SC15: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 15 - 20, 2015

Texas, Austin

Acceptance Rates

SC '15 Paper Acceptance Rate 79 of 358 submissions, 22%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

52
Total Citations
View Citations
974
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)18

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang WDong YCrempien JArduino PKurtulus ATaciroglu E(2024)A comparison of ground motions predicted through one-dimensional site response analyses and three-dimensional wave propagation simulations at regional scalesEarthquake Spectra10.1177/8755293024123193540:2(1215-1234)Online publication date: 28-Feb-2024
https://doi.org/10.1177/87552930241231935
Gill ALalith MHori MOgawa Y(2024)Analysis of postdisaster economy using high‐resolution disaster and economy simulationsRisk Analysis10.1111/risa.17662Online publication date: 9-Oct-2024
https://doi.org/10.1111/risa.17662
Wan JWang WZhang Z(2024)Enhancing computational efficiency in 3-D seismic modelling with half-precision floating-point numbers based on the curvilinear grid finite-difference methodGeophysical Journal International10.1093/gji/ggae235238:3(1595-1611)Online publication date: 5-Jul-2024
https://doi.org/10.1093/gji/ggae235
Fu JBa ZLiang JWang F(2024)A Physics‐Based Simulation Approach for Urban‐Scale Earthquake Disaster From Fault to City: Theory, Verification, and ApplicationEarthquake Engineering & Structural Dynamics10.1002/eqe.4278Online publication date: 28-Nov-2024
https://doi.org/10.1002/eqe.4278
Faj JKenter TFaghih-Naini SPlessl CAizinger VHuebl ASilvano CRobinson T(2023)Scalable Multi-FPGA Design of a Discontinuous Galerkin Shallow-Water Model on Unstructured MeshesProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3592979.3593407(1-12)Online publication date: 26-Jun-2023
https://dl.acm.org/doi/10.1145/3592979.3593407
Wan WGan LWang WYin ZTian HZhang ZWang YHua MLiu XXiang SHe ZWang ZGao PDuan XLiu WXue WFu HYang GChen XSong ZChen YLiu XZhang WMohror KArnold DBadia R(2023)69.7-PFlops Extreme Scale Earthquake Simulation with Crossing Multi-faults and Topography on SunwayProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613209(1-15)Online publication date: 12-Nov-2023
https://dl.acm.org/doi/10.1145/3581784.3613209
Zhang WCrempien JKurtulus AChen PArduino PTaciroglu E(2023)A suite of broadband physics‐based ground motion simulations for the Istanbul regionEarthquake Engineering & Structural Dynamics10.1002/eqe.380952:4(1161-1181)Online publication date: 7-Jan-2023
https://doi.org/10.1002/eqe.3809
Touhami SGatti FLopez-Caballero FCottereau Rde Abreu Corrêa LAubry LClouteau D(2022)SEM3D: A 3D High-Fidelity Numerical Earthquake Simulator for Broadband (0–10 Hz) Seismic Response Prediction at a Regional ScaleGeosciences10.3390/geosciences1203011212:3(112)Online publication date: 2-Mar-2022
https://doi.org/10.3390/geosciences12030112
Motoyama HHori M(2022)Construction and Usefulness Verification of Modeling Method of Subsurface Soil Layers for Numerical Analysis of Urban Area Ground MotionGeoHazards10.3390/geohazards30200133:2(242-251)Online publication date: 9-May-2022
https://doi.org/10.3390/geohazards3020013
Ichimura TFujita KKoyama KKusakabe RKikuchi YHori THori MMaddegedara LOhi NNishiki TInoue HMinami KNishizawa STsuji MUeda N(2022)152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulationInternational Conference on High Performance Computing in Asia-Pacific Region10.1145/3492805.3492814(18-29)Online publication date: 7-Jan-2022
https://dl.acm.org/doi/10.1145/3492805.3492814
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents